• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

Pro tip: If you choose to announce at a conference where many other vendors will surely announce news also, you naturally run the risk of not garnering much attention.

For starters, it may help to realize or recall that:

• Cassandra was originally developed and revealed at Facebook, to much early NoSQL fanfare. Facebook later backed away from Cassandra use.
• Rackspace guys in Texas became Cassandra’s biggest backers. They eventually founded a company called Riptano to commercialize Cassandra.
• Texas company Riptano became the California company DataStax.
• DataStax came out with a Hadoop-on-Cassandra offering called Brisk. For a while, it sounded as if Hadoop was as big a focus for DataStax as Cassandra is.
• DataStax is now recommitted to being the Cassandra company, and has accordingly backed away from Hadoop and Brisk as a separate or coequal focus. However, it sees Hadoop capability as a nice, or even major, feature of its Cassandra-centric offering.
• To finalize its open source obligations with respect to Brisk, DataStax is in essence:
  • Donating a Hive driver for Cassandra straight into the main Apache Cassandra project.
  • Releasing the rest of Brisk as a separate open source project.
  • Disclaiming interest in further advancing open source Brisk.
• There’s also something called Solandra — evidently SOLR-on-Cassandra — whose status is similar to Brisk’s.
• There are three main ways that DataStax helps you to consume Cassandra.
  • DataStax is the principal sponsor of Apache Cassandra development, and presumably long will be. Apache Cassandra is both free-like-speech and free-like-beer.
  • DataStax is also introducing a paid-subscription version of Cassandra called DataStax Enterprise, which features proprietary code, support, and so on. DataStax Enterprise is neither free-like-speech nor free-like-beer.
  • There will also be something called DataStax Community Edition. DataStax Community Edition is free-like-beer, but not free-like-speech.

Various posts on the DataStax blog give DataStax’s explanation of what it’s doing. Ben Werther, the ex-Greenplum guy who briefly worked at DataStax and was most associated with telling the Hadoop/Brisk story, has moved on to his own startup Platfora.

DataStax Enterprise has three main aspects:

• DataStax Server, which is the actual database and analytics code. At this time, there is little closed-source code in DataStax Server, but DataStax reserves the right to widen that gap in the future.
• DataStax OpsCenter, which is management tools around DataStax Server. DataStax OpsCenter is entirely closed-source, even though DataStax gives a limited version away for free.
• Support.

To describe DataStax Community Edition, I’ll just quote the press release verbatim, which characterizes it as:

… a free platform based on Apache Cassandra that bundles the open source database with smart installers, drivers and connectors for popular development languages, demo apps, documentation, and a free version of DataStax OpsCenter for Apache Cassandra.

DataStax Community Edition is crippleware only in terms of feature set; there are no limitations on its database size, cluster size, or usage rights. A core mission of DataStax Community Edition is to create happy Cassandra users, who may then become customers for DataStax Enterprise.

• 100s of gigabytes of data at first, growing to >1 terabyte over time.
• High peak loads.
• Public cloud portability (but they have private data centers they can use today).
• Simple database design — not a lot of tables, not a lot of columns, not a lot of joins, and everything can be distributed on the same customer_ID key.
• Stream the data to a data warehouse, that will grow to a few terabytes. (Keeping only one year of OLTP data online actually makes sense in this application, but of course everything should go into the DW.)

So I’m leaning to saying:  

• They should go with a scalable, MySQL-based solution.
  • Lots of third-party software works with MySQL, in case that’s helpful.
  • Yes, any one vendor is small and not yet firmly established, but there are numerous vendors around with interesting MySQL scaling stories.
  • In a vendor emergency, just going with Oracle’s MySQL stuff would probably work …
  • … especially because there are these lovely things in the world called solid-state drives.
  • There’s also good escapability if one wants to move away from MySQL, because everybody knows how to handle MySQL data.
• The first product to look at is dbShards, because it meets all the topology needs:
  • Local scale-out (transparent sharding).
  • Local high availability.
  • Remote disaster recovery (details of that are underway).
• The first analytic DBMS to look at is Infobright.
  • Yes, I know Infobright is focused more on machine-generated data these days, but this client’s analytic needs are so straightforward Infobright should pass with flying colors.
  • The MySQL-to-MySQL aspect should make ETL dead simple.
  • Again, there’s escapability.

Mainly, this is all fine. But I’m getting pushback on the solid-state aspect, for fear that it will compromise public cloud portability.

Am I missing something here? As far as I’m concerned, if you’re planning an OLTP system with a many-year lifespan today, of course you should assume solid-state storage. Maybe you scale out just as far as you would with disk, striping indexes or entire databases across the RAM of multiple servers. It that case, having solid-state backing reduces the risk of bottlenecks. Maybe you don’t scale out as far as you would with disk. In that case, solid-state backing saves you money.

As for public-cloud support for solid-state storage, that’s coming fast, right? (Actually, I have data points in support of that theory, but they’re a bit tenuous.) A large fraction of web businesses with private data centers seem to be using solid-state storage — from Facebook on down — or so the NoSQL/NewSQL/short-request DBMS guys tell me. Surely a number of public cloud vendors are close behind.

After Michael Stack of StumbleUpon beat me up for a while,* Omer Trajman of Cloudera was kind enough to walk me through HBase usage. He is informed largely by 18 Cloudera customers using, plus a handful of other well-known HBase users such as Facebook, StumbleUpon, and Yahoo. Of the 18 Cloudera customers Omer was thinking of, 15 are in HBase production, one is in HBase “early production”, one is still doing R&D in the area of HBase, and one is a classified government customer not providing such details.

*Just kidding — he was actually extremely gentle.

In the use cases that Omer offered, what’s stored in HBase is almost always records of web or network activity. Specific examples included clickstream information (at 5 different ad companies), crash reports (at Mozilla), and messages (at Facebook). Sometimes the data gets into Hadoop twice — once excerpted via HBase and once as part of a full log — and may even live in two different Hadoop clusters.

What’s served out from HBase in Omer’s examples is usually derived data, such as a user profile, an ad selection, a text index, etc. That makes sense, not least because if you’re going to keep enhancing your data, schema-free programming — which HBase offers — looks ever more appealing. Omer further said that there are a growing number of cases in which HBase is being used to serve up reference data for batch MapReduce jobs, but he didn’t have specifics. A counterexample to the derived data emphasis would be, if I understood correctly, a case where HBase manages shopping carts.

I haven’t put much effort into unearthing open source or other third-party HBase-based projects, but two examples are Open  TSDB  (Time Series DataBase) and Lily CMS (Content Management Systems). (Edit: But see the comment about Lily below.)

Omer is perhaps my top go-to guy on database and cluster sizes, so of course I asked him for HBase metrics as well. He responded (approximately) that Cloudera HBase customer installations average 20-30 nodes, but that half a dozen are in the 100-200 node range.

Finally, there’s the matter of latency. As a general rule, the HBase users Omer sees are using HBase with at least several minutes latency. (Again , that shopping cart case would seem to be a counterexample.) So, for example, the data recorded when you click on a page isn’t immediately applied toward tweaking your profile to determine which ad you’ll see next — but it might come into play after you spend a few minutes reading the page you’re on. Naturally, Omer knows of efforts to use HBase with lower latency yet, and I won’t be surprised if already-working examples of low-latency HBase show up in the comment thread to this post.

• Facebook et al. are in effect Software as a Service (SaaS) vendors, not enterprise technology users. In particular:
  • They have the technical chops to rewrite their code as  needed.
  • Unlike packaged software vendors, they’re not answerable to anybody for keeping legacy code alive after a rewrite. That makes migration a lot easier.
  • If they want to write different parts of their system on different technical underpinnings, nobody can stop them. For example …
  • …  Facebook innovated Cassandra, and is now heavily committed to HBase.
• It makes little sense to talk of Facebook’s use of “MySQL.” Better to talk of Facebook’s use of “MySQL +  memcached  + non-transparent sharding.” That said:
  • It’s hard to see why somebody today would use MySQL +  memcached  + non-transparent sharding for a new project. At least one of Couchbase or transparently-sharded MySQL is very likely a superior alternative. Other alternatives might be better yet.
  • As noted above in the example of Facebook, the many major web businesses that are using MySQL +  memcached  + non-transparent sharding for existing projects can be presumed able to migrate away from that stack as the need arises.

Continuing with that discussion of DBMS alternatives:

• If you just want to write to the memcached API anyway, why not go with Couchbase?
• If you want to go relational, why not go with MySQL? There are many alternatives for scaling or accelerating MySQL — dbShards, Schooner, Akiban, Tokutek, ScaleBase, ScaleDB, Clustrix, and Xeround come to mind quickly, so there’s a great chance that one or more will fit your use case. (And if you don’t get the choice of MySQL flavor right the first time, porting to another one shouldn’t be all THAT awful.)
• If you really, really want to go in-memory, and don’t mind writing Java stored procedures, and don’t need to do the kinds of joins it isn’t good at, but do need to do the kinds of joins it is, VoltDB could indeed be a good alternative.

And while we’re at it — going schema-free often makes a whole lot of sense. I need to write much more about the point, but for now let’s just say that I look favorably on the Big Four schema-free/NoSQL options of MongoDB, Couchbase, HBase, and Cassandra.

Cloudera can identify 22 CDH (Cloudera Distribution [of] Hadoop) clusters holding one petabyte or more of user data each, at 16 different organizations. This does not count Facebook or Yahoo, who are huge Hadoop users but not, I gather, running CDH. Meanwhile, Eric Baldeschwieler of Hortonworks tells me that Yahoo’s latest stated figures are:

• 42,000 Hadoop nodes …
• … holding 180-200 petabytes of data.

That works out near the low end of the range I came up with for Yahoo’s newest gear, namely 36-90 TB/node. Yahoo’s biggest clusters are little over 4,000 nodes (a limitation that’s getting worked on), and Yahoo has over 20 clusters in total.

Based on those numbers, it would seem that 10 or more of Yahoo’s Hadoop clusters are probably in the petabyte range. Facebook no doubt has a few petabyte-scale Hadoop clusters as well. So we’re probably over 3 dozen petabyte+ Hadoop clusters, just counting Yahoo, Facebook, and CDH users. There surely are others too, running Apache Hadoop without Cloudera’s help.

We also have some more information about the scale of Hadoop usage, and the markets it is being used in, because Omer Trajman of Cloudera kindly wrote the following — lightly edited as usual — for quotation:

The number of Petabyte+ Hadoop clusters expanded dramatically over the past year, with our recent count reaching 22 in production (in addition to the well-known clusters at Yahoo! and Facebook). Just as our poll back at Hadoop World 2010 showed the average cluster size at just over 60 nodes, today it tops 200. While mean is not the same as median (most clusters are under 30 nodes), there are some beefy ones pulling up that average. Outside of the well-known large clusters at Yahoo and Facebook, we count today 16 organizations running PB+ clusters running CDH across a diverse number of industries including online advertising, retail, government, financial services, online publishing, web analytics and academic research. We expect to see many more in the coming years, as Hadoop gets easier to use and more accessible to a wide variety of enterprise organizations.

Omer went on to add:

The biggest number of PB clusters are in the advertising space. I often tell people that every ad you see on the internet touched at least one Hadoop cluster (or the Google equivalent).

*Not coincidentally, two of the oldest names for “IT” were data processing and management information systems.

Eventually, there came to be three views of the essence of IT:

• Data – i.e., the traditional view, still exemplified by IBM and Oracle.
• People empowerment — i.e., Microsoft-style emphasis on UI friendliness and efficiency.
• Operational workflow — i.e., SAP-style emphasis on actual business processes.

Graphical user interfaces were a major enabling technology for that evolution. Equally important, relational databases made some difficult problems easy(ier), freeing application designers to pursue more advanced functionality.

Based on further technical evolution, specifically in analytic and consumer technologies, I think we should now take that list up to five. The new members I propose are:

• Investigative analytics.
• Emotional response.

I may want to rename that last one someday, but “emotional response” will serve well enough for now.

At first blush, it might seem that investigative analytics could be regarded as straightforward database processing, perhaps based on some analytics-friendly data structure; but that’s not true today, and I can’t pinpoint a past era when I think it was true either. Defining and structuring the data is just a starting point, and is not tantamount to saying what will be done with it.* For example, exactly the same data mart might be used for:

• Conventional business intelligence —  reporting, query, charting and drill-down (“multi-dimensional” or otherwise).
• Similar activities, but with a QlikView or Endeca kind of spin.
• Statistics and machine learning.
• Graphical analysis.

Perhaps the best example of an investigative-analytics-oriented company is Google, where nothing is acknowledged as true until it’s been statistically verified, and whose core service is the world’s most flexible research tool.

*True, in some cases the introduction of derived/cooked data could lead the data structure to evolve in an application-specific way. But I don’t think that seriously undermines the basic point.

Meanwhile, as Ray Wang likes to point out, a large fraction of current IT innovation is consumer-centric. I’m not sure I’d grant all his business-to-business sub-points to that claim, but it surely is true that:

• Consumer-oriented technology is much more sophisticated than it used to be.
• A large fraction of what’s done in information technology is consumer- or at least customer-facing.

The key defining trait of consumer/customer (as opposed to employee) computer use is that it is optional; your customers don’t have to do business with you, or use the applications and interfaces you offer. Consequently, it’s your responsibility to make sure that what you ask them to do is convenient and engaging (or fun, or at least not-annoying). And so, much more than before, you need to be concerned about users’ emotional reactions to computing systems. Personalization fits in with both the investigative analytics and emotional-engagement themes. Investigative analytics is why you can personalize people’s experiences (web surfing, shopping, gaming, etc.); emotional response is why you must. Companies I’d put in the “emotional” camp include Facebook, whose central concept is “friend”; Zynga, whose customer-retention strategy seemingly boils down to addiction; and — back in its heyday — AOL.

Related link

• Three kinds of software innovation

Fusion-io is growing very, very fast, doubling or better in revenue every 6 months.

Fusion-io’s marketing message revolves around “data centralization”. Fusion-io is competing against storage-area networks and storage arrays.

Fusion-io’s list of application types includes

… systems dedicated to decision support, high performance financial analysis, web search, content delivery and enterprise resource planning.

Fusion-io says it has shipped over 20 petabytes of storage.

Fusion-io has a shifting array of big customers, including OEMs: 

Historically, large purchases by a relatively limited number of customers have accounted for a substantial majority of our revenue, and the composition of the group of our largest customers changes from period to period. Many of our customers make concentrated purchases to complete or upgrade specific large-scale data storage installations. These concentrated purchases are short-term in nature and are typically made on a purchase order basis rather than pursuant to long-term contracts. During fiscal 2010 and the six months ended December 31, 2010, sales to the 10 largest customers in each period, including the applicable OEMs, accounted for approximately 75% and 92% of revenue, respectively. Facebook, Inc. is currently our largest customer and accounted for a substantial portion of revenue during the six months ended December 31, 2010. We expect revenue from sales to Facebook and one other end-user to account for a substantial portion of revenue for the three months ending March 31, 2011, but that revenue from sales to Facebook and the other end-user will decline significantly for the three months ending June 30, 2011 as they complete their planned deployments.

But Fusion-io invests enough in sales and marketing, including direct sales, that I’m guessing they’re out there persuading end-users to ask for product from Dell, HP, and IBM.

Fusion-io’s inventory growth of $23.3 million for the second half of 2010 is close to revenue of $26.0 million. Accounts receivable is a much smaller figure. I’m not sure what all that signifies, but I do find it ironic that Fusion-io’s marketing statements draw an analogy to “just-in-time” manufacturing.

As for what I think about Fusion-io, it starts:

• Fusion-io’s ideas are smart.
• My skepticism about specialized storage hardware for database applications applies in part but not in whole to Fusion-io.
• Right now, Fusion-io has won the market. Even if you don’t need Fusion-io hardware to optimize your use of solid-state memory, you’re apt to go with/partner with Fusion-io anyway.

I don’t have strong opinions as to how long the last point will remain true.

There are two technology trends driving electronic privacy threats. Taken together, these trends raise scenarios such as the following:

• Your web surfing behavior indicates you’re a sports car buff, and you further like to look at pictures of scantily-clad young women. A number of your Facebook friends are single women. As a result, you’re deemed a risk to have a mid-life crisis and divorce your wife, thus increasing the interest rate you have to pay when refinancing your house.
• Your cell phone GPS indicates that you drive everywhere, instead of walking. There is no evidence of you pursuing fitness activities, but forum posting activity suggests you’re highly interested in several TV series. Your credit card bills show that your taste in restaurant food tends to the fatty. Your online photos make you look fairly obese, and a couple have ashtrays in them. As a result, you’re judged a high risk of heart attack, and your medical insurance rates are jacked up accordingly.
• You did actually have that mid-life crisis and get divorced. At the child-custody hearing, your ex-spouse’s lawyer quotes a study showing that football-loving upper income Republicans are 27% more likely to beat their children than yoga-class-attending moderate Democrats, and the probability goes up another 8% if they ever bought a jersey featuring a defensive lineman. What’s more, several of the more influential people in your network of friends also fit angry-male patterns, taking the probability of abuse up another 13%. Because of the sound statistics behind such analyses, the judge listens.

Not all these stories are quite possible today, but they aren’t far off either.

One of the supporting trends, pretty obvious, is that there is a lot more electronic information than there used to be. Indeed:

• Sufficient information exists to provided a very detailed picture of our activities.
• Much of it is recorded for very good and beneficial reasons. We wouldn’t want that  part to stop.
• This information is inevitably available to government.

Here’s what I mean by the inevitability claim. Whether or not you think anti-terrorism concerns are overblown, as a practical matter your fellow voters* will allow a broad range of governmental information access. Besides, just the widely-available credit card and similar commercial data is enough to provide a fairly detailed picture of what you’re up to. In most countries, anti-pornography, anti-file-sharing, and/or general civilian law enforcement efforts serve to strengthen the point further.

*If you live in a country too unfree for voters to much matter, then it is surely also the case that governmental information has few practical limits.

Examples of information being tracked (more particulars were covered in the first post of this series):

• Almost everything we buy is recorded, via credit card transactions, point-of-sale data, and/or website transaction records. This data is summarized in files covering 100s of millions of individuals, with 1000s of fields per person. Those files can be used for a broad variety of business or law enforcement purposes.
• That data gives a great picture of what we eat, where we commute or travel, what we pay attention to, and so on.
• All our other financial information also passes through computer systems, such as at banks.
• Increasingly, our physical movements are tracked more directly, via cell phones (our own), police cameras, and the like.
• Other than face-to-face conversations, almost all our communications are electronic. Even social media non-adopters rely heavily on telephones, email, and the like.
• Increasingly, our reading and viewing entertainment choices are electronically recorded as well.

Most of that data is available to law enforcement departments. Much of it is available to commercial companies as well.

And these vast amounts of data will hardly go to waste. The second major technological trend in play is that the data can be much more effectively analyzed than before. New kinds of or effectiveness in analytic profiling create whole new levels of exposure (using the word “exposure” in its most literal sense), in at least three ways:

• Relationship profiling.* Relationship analytics technology has been around for a while. When it’s used to find bad guys (terrorists, fraudsters, etc.), that’s one thing. But some of the marketing uses are spookier. Marketing-like uses applied back to governmental surveillance could be spookier yet.
• Propensity profiling.* A huge fraction of what happens in big data analytics is figuring out what you’re likely to buy, vote for, look at, click on, react to, or think. Marketers getting that right can be a bit creepy. So can marketers getting it wrong. Governments doing the same thing could be much creepier yet.
• De-anonymization.* You may think you can be anonymous online, but you really can’t. Also, it’s getting ever harder to keep your roles or activities online separate from each other.

* I just coined the terms “relationship profiling” and “propensity profiling.” “De-anonymization,” however, has been in use for a while.

Classical relationship profiling questions include assessing who has a close relationship with whom, who influences whom, who influences lots of people, etc. The most obvious data to infer this from is communication — who called whom, how long they talked, who they called next, what time of day this all happened, and so on. Anti-terrorist uses are obvious. A major marketing use is telcos — who of course have this data — deciding who to offer their best deals to, by trying to identify who influences the most other customers. These calculations of course involve comparing lots of data, mainly about people who are NOT targets of terrorist investigation or preferential telephone service pricing.

Much of Facebook’s $50 billion valuation hinges on the assumption it can do similar things based on the “social graph” it infers from informal communication among friends. To date that assumption has been questionable, but we’re still in the very early days. Meanwhile, cruder methods of analyzing social influence are used. But the trend is clear — marketers want to use technology to identify social leaders, influence them however they can, and hope that the rest of us follow along baaing. Up to a point, that’s actually OK — learning things from our friends and acquaintances is an important and pleasant part of living in a society. And political campaigners have been doing it for generations, in the most low-tech of fashions. Still, it’s one thing for such targeting of leaders to be transparent; if done surreptitiously, it suddenly starts to feel a lot more sinister.

For years, propensity profiling has been an area of huge investment and technological progress. It’s the central application of big data analytics, and the heart of the business for many companies I write about, or that are my clients. Credit files, web logs, other marketing responses, census information, and other data are combined to infer:

• Your income, household composition, age, race, education, and other basic demographics.
• Your buying, voting, reading, viewing, and other consuming interests in minute psychographic detail.
• Your feelings about particular companies and brands, your propensity to become or stop being their customer, and what kinds of advertisements or offers it would take to influence you.
• Your status as a credit risk.
• The chance you are committing or will commit fraud.

This has been going on since at least the 1990s, especially in service industries with “loyalty card” kinds of programs, such as retail or travel/leisure. In the credit case it’s been going on longer than that. But new data sources, processed by new analytic technologies, have brought the practice to a vastly greater height.

Finally — in case you care about being anonymous online, you’re running out of luck. De-anonymization analytics are getting too good. The Electronic Freedom Foundation’s de-anonymization overview in 2009 was one of many articles pointing out that it often was possible to attach a specific name to online activities that in theory don’t track personally identifiable information. Meanwhile, at a talk I attended in May, 2010, comScore spoke of its successful efforts to tie various anonymous online activities, such as visits to different websites, to each other. And after I entered “usinger.com” into my browser address bar, I started seeing ads for Usinger sausages at a variety of prominent websites.

I’m not sure how much of a privacy threat de-anonymization technology is in and of itself, but it certainly provides support to both relationship and propensity profiling.

The privacy discussion has gotten more active, and more complicated as well. A year ago, I still struggled to get people to pay attention to privacy concerns at all, at least in the United States, with my first public breakthrough coming at the end of January. But much has changed since then.

On the commercial side, Facebook modified its privacy policies, garnering great press attention and an intense user backlash, leading to a quick partial retreat. The Wall Street Journal then launched a long series of articles — 13 so far — recounting multiple kinds of privacy threats. Other media joined in, from Forbes to CNet. Various forms of US government rule-making to inhibit advertising-related tracking have been proposed as an apparent result.

In the US, the government had a lively year as well. The Transportation Security Administration (TSA) rolled out what have been dubbed “porn scanners,” and backed them up with “enhanced patdowns.” For somebody who is, for example, female, young, a sex abuse survivor, and/or a follower of certain religions, those can be highly unpleasant, if not traumatic. Meanwhile, the Wikileaks/Cablegate events have spawned a government reaction whose scope is only beginning to be seen. A couple of “highlights” so far are some very nasty laptop seizures, and the recent demand for information on over 600,000 Twitter accounts. (Christopher Soghoian provided a detailed, nuanced legal analysis of same.)

At this point, it’s fair to say there are at least six different kinds of legitimate privacy fear. The first five I have in mind are:

• Governmental force. Your web browsing history can be used against you in a court of law. Profiling — perhaps based on big data analytics — might make you a target for law enforcement or anti-terrorism investigation as well. And between financial transaction records, communication records, physical movement data, and more, the government’s ability to gather and process information about people is effectively unlimited.
• Private sector discrimination. There are many ways private companies could use detailed profiles — or just incriminating photos — to your detriment. They could fire you, not hire you, deny you insurance or credit, or simply not give you their best possible price.
• Identity theft. Phishing happens.
• Social pressure and stalking. What your friends, neighbors, and classmates know about you can become a serious problem, especially if they gang up and cyber-bully you.  That your violent ex-boyfriend can track you could be a bigger problem yet.
• Embarrassment and creep-out. Some people REALLY don’t like being viewed naked in busy public places. Some people are bothered by weirdly (in)appropriate advertisements. Maybe there’s no obvious tangible harm other than their uncomfortable feelings — but their discomfort is serious even so.

The sixth is:

• Throwing out the baby with the bathwater in a backlash. I think medical privacy rules already cost lives, in research and treatment alike. (Forbes offers multiple examples of life-saving research being stopped by HIPAA.) I think there’s an inevitable tradeoff between intrusive physical security measures, such as the TSA’s various forms of security theater, and spooky behind-the-scenes profiling and surveillance. (Unfortunately, we can hardly live in a society without some kinds of security measures — and even if we could, voters would never allow it.) Proposals that would hamstring the internet advertising industry currently seem unlikely to pass — but how much uproar would it take before that changed?

You probably knew most of that already. Even so, here are a number of examples and links.

Slashdot has more on the Jensen case, in which a man was convicted of murder in no small part because his computer revealed that he had searched for information about murder methods, including one that duplicated the actual method of his wife’s death. The money quote in an appeals court decision is in Section 37.1, which characterizes “computer evidence” as “probably the most incriminating other evidence.”

Obviously, that particular trial had the correct outcome. But would you want a court to hear about the research you did about minimizing your taxes? What about when you considered changing jobs, something that might be of interest both in employment and child custody litigation? How about your viewing of sexy images other than those of your spouse? And by the way — just what kinds of online viewing or writing should get you on a terrorist watch list?

It’s getting ever more practical to track our actions and movements. The privacy implications are potentially grave — would you want THAT kind of information used in court, or for decisions about your insurance? Accordingly, the Electric Freedom Foundation coined the word “traitorware” to describe and call attention to consumer (mainly) devices that keep or even transmit records of your movements and doings. And mind-bogglingly, Forbes says that Sprint “turned customers’ GPS information over to law enforcement 8 million times in a year.”

But it’s not just your own devices. The New York Times recently wrote of uses for smart video technology. Enforcing good hand-washing procedure on doctors sounds great. But how about enforcing busy-ness on cubicle workers? Watching movie previewers’ emotional reactions to specific scenes and characters seems kosher. But what if all movie-goers are watched? Or what if this is done at supermarkets or vending machines, perhaps even repeatedly over time? Could be creepy, huh? And by the way, while it could be a way to get you great discounts and service, it could also be a way to categorize you as unworthy of same.

Meanwhile, the Washington Post reminds us that battlefield video surveillance technology is getting ever better. Obviously, such technology could be used for domestic law enforcement as well. More immediately, the Washington Post tells us of things like automatic recognition of license plates being used by police departments that repurpose anti-terrorism grants to general law enforcement. (Analogies to the 1980′s habit of tying every grant proposal to the Strategic Defense Initiative “Star Wars” project are probably not misplaced.)

But the benefits foregone if we don’t use this technology are also scary. The biggest current examples are probably the ones I cited above — medical research, anti-terrorism, and the whole online advertising industry. But even more examples are coming down the pike, of clever electronic aids with privacy implications. Most notably — as the population ages, and nursing homes continue to be miserable (and costly) places, revolutionizing elder care becomes ever more desirable. Well, sensor technology will soon be able to watch over old folks, keeping them safe(r) in their independence — but only if it is highly detailed or intrusive.

Also:

• Slashdot/New York Times had an article on a Marin County ordinance forbidding smart electrical meters. Part of the reason is a San Francisco area backlash against the privacy implications of having your electricity usage recorded moment to moment.
• Slashdot also points us at an article about technology you use for self-control in various ways. In principle, however, the “self” part is an optional feature.
• Computing Now offered an overview of on-body sensing technology. Mainly it talks about gaming and medical applications, but it also suggests that people receptive to putting tracking technology on themselves in crowds in return for certain emergency-response benefits.

Finally, privacy-threatening observations of our web use, postings, and other internet communications are much too numerous to list. But some high/lowlights include:

• AddThis now claims to track 1 billion unique individuals online.
• In its transparency reports, Google publishes a count of how often various governments ask it for data. For the first half of 2010, leaders included the US (4287), Brazil (2435), India (1420), the UK (1343), and France (1017). Also over 100 requests each were Argentina, Australia, Chile, Germany, Italy, Singapore, Spain, and Taiwan.
• Verizon had over 90,000 such requests in 2007.
• An internet service provider who successfully fought off an FBI information request nonetheless was kept under a gag order for six years about the case.
• The Obama Administration wants ever more new legal powers to get electronic information without court order or notice to the investigation’s subjects.
• The UK government, or parts thereof, wants to track everything you do online or telephonically, except — supposedly — the actual contents of your calls and messages. Similarly Australia. There’s an EU directive along those lines as well, although the EU also has some strong data retention safeguards.
• Various governments — India, the United Arab Emirates, and many more — have recently insisted that communication offerings such as BlackBerry provide them with back doors to snoop, on pain of being banned countrywide.
• Informal methods of data acquisition are also used. For example, police departments use fake identities (with photos of hot girls, of course) to friend young folks on Facebook to look for photos of underage drinking.
• A CNN survey article showed how extreme public data revelation can get, using the example of Louis Gray.
• For its own marketing reasons, DuckDuckGo offers a simple overview of how your web searches could/can be used against you.
• And as an example of how even petty internet uses can cause problems, one of many things that got former HP CEO Mark Hurd into trouble was looking at racy online pictures of business associate Jodie Fisher.