• Data preparation, which is tedious unless you do a good job of automating it.
• Running the actual algorithms.

Numerous statistical software vendors (or open source projects) help you with the second part; some make strong claims in the first area as well (e.g., my clients at KXEN). Even so, large enterprises typically have statistical silos, commonly featuring expensive annual SAS licenses and seemingly slow-moving SAS programmers.

As I see it, the predictive analytics workflow goes something like this:

• Business-knowledgeable people develop a theory as to what kinds of information and segmentation could be valuable in making better business micro-decisions.
• Statistics-knowledgeable people determine a structure for modeling that reflects this theory.
• Statistics-knowledgeable people tweak the model over time, within a fixed general structure, as new data comes in.
• (Optional) Somebody sees to acquiring whatever data is needed that the organization doesn’t already have (and won’t get in the ordinary course of ongoing business).

The optional last part can be a purchase of third-party information (relatively fast and easy) or the development of a business process (and if necessary associated software) to capture the information (not always so easy). But even if that’s taken care of, or not present, we have at least two hand-offs where agility can be lost:

• Businesspeople may throw a request “over the wall” to the statisticians, who then work on it as their schedule permits.
• Once created, a model may be so set in stone that even small changes are as hard as building a new model from scratch.

The second problem can be solved by the statisticians themselves, without outside involvement. Model research and model refinement should be separate processes. You can recheck your clustering on one schedule, but recalibrate your regressions against each cluster more frequently. If that all sounds forbiddingly difficult, perhaps your model recalibration process needs another level of automation.

So I’ve finally gotten to the point of saying what may have been obvious from the start: The only excusable impediment to predictive analytic agility is the hand-off from the people who know the business to the people who know the math. So let’s examine ways that difficulty can be resolved.

At big internet companies, the usual answer is something like

Hey, it’s just data. From web logs. And network event logs. The data scientists know how to handle that.

In financial trading firms, the answer is more

The traders and analysts work closely together. Very closely. In fact, when the traders rip out their phones and throw them across the room, the analysts need to duck to avoid getting clobbered.

In credit card or telecom marketing or insurance actuarial organizations, the answer may be

Don’t worry; the stats geeks have been at this for a long time; they really do understand our business.

All three approaches work.

But what about conventional enterprises, where line-of-business people may not be as math-savvy as internet developers or financial traders, and where the math experts may not have the business issues down cold? My flippant answer is that businesspeople should know some math too.* My more serious answer is that the “business analyst” role should be expanded beyond BI and planning to include lightweight predictive analytics as well.

*I wasn’t being entirely flippant, of course. Statistics is even being taught in high school these days. And when I got a PhD in game theory, 2/3 of my thesis committee was at the Harvard Business School.

For example, at retailers:

• Market basket analysis is pretty simplistic (it only looks at small subsets of a basket at a time).
• Seasonality is tricky. (Weather and so on can skew it.)
• Each store or region can be its own universe.
• Some of the results of analytics are rather coarse-grained — e.g., merchandise adjacencies — so precision in statistical analysis may not matter much anyway.

And so truly rigorous statistical analysis may be both unfeasible and unnecessary; a lot of business-informed seat-of-the-pants reasoning needs to be mixed in. Consequently, there’s a lot to be said for pushing at least some retail predictive analytics pretty close to the merchandising department(s).

Similar stories could be told in many other industries and pursuits, including but emphatically not limited to:

• Event marketing.
• College admissions.
• Political campaigning.
• Field maintenance at utility companies.
• Price-setting (across many industries).

In each case, it’s easy to see how statistical and predictive analytic techniques could add real value to the business. But it’s hard to imagine how the enterprise could support the kind of large, experienced, business-knowledge analytic operation one might find in hedge fund investing or telecom churn analysis. And absent that, it’s tough to see why the only people doing predictive analytics for the organization should sit in some silo of statistical expertise.

10gen recommends solid-state storage in many cases. In some cases solid-state lets you get away with fewer total nodes. 10gen also likes Flashcache (Facebook-developed technology to put a flash cache in front of hard disks). But the 100-node example mentioned above uses spinning disk.

Use cases 10gen is proud of include:

• Lots of user profile maintenance, including at online ad companies. This includes full user ad impression data. (I’ve argued for a while that user profile information belongs in something like a NoSQL database.)
• A big-name web company that wants to inspect every packet that enters their network, and replaced Splunk with MongoDB for performance reasons.
• A big-name photo/video site whose metadata is all in MongoDB. (That’s the kind of thing that often makes for good MarkLogic use cases.)

But actually, the reason we had the call was to review cases where MongoDB’s schemaless nature was significant. Examples of those included:

• A couple of top examples were of the kind “A bunch of apps, similar but not the same.” For MTV, it’s a single content management system for a bunch of websites. For Disney Playdom, it’s different schemas for every game.
• For a wireless telco, the issue was a product catalog in which devices and service plans called for very different schemas, and which the telco felt had thus become unmanageable in Oracle.
• For Craigslist, the issue wasn’t programming so much as performance — ALTER TABLE operations took months in MySQL, and that’s not a typo, although I’ll confess to not understanding why this was the case.

The 10gen guys went on to claim that schemalessness is helpful for incremental development in general, the point being that you don’t have a database-modification step. To some extent, changes can even be rolled back more easily than if you actually changed your schemas.

*I could be wrong in those guesses; as small companies go, McObject is unusually prone to secrecy games.

As best I understand:

• eXtremeDB is something like an in-memory object-oriented DBMS, designed to be embeddable.
• However, much as with Objectivity and other old-school OODBMS, eXtremeDB winds up being more of a toolkit with which to build DBMS than a full DBMS.
• eXtremeDB has a few indexing schemes. The main one is good old B-trees. One customer wanted Patricia tries, so they’re in there. (Perhaps not coincidentally, solidDB relies on Patricia tries.) At least one wanted R-trees, so they’re in there too.
• eXtremeDB has long had the option of persistent logs.
• eXtremeDB newly has a hybrid memory-centric option, in which you can have more data in the database than fits into RAM.
• eXtremeDB newly has multi-master two-phase-commit clustering.

My guess three years ago that eXtremeDB might emerge as an alternative to solidDB seems to have been borne out. McObject CEO Steve Graves says that the core of McObject’s business is OEMs, in sectors such as telecom equipment and defense/aerospace. That’s exactly solidDB’s traditional market, except that solidDB got acquired by IBM and deemphasized it.

I’ve said before that if I were starting a SaaS effort — and it wasn’t just focused on analytics — I’d look at using a memory-centric OODBMS. Perhaps eXtremeDB is worth looking at in such scenarios.

Bit bucket

• Kinds of data likely to be included: Logs, other technical/external
• Likely use styles: Staging/ETL, investigative
• Canonical example: Log files in a Hadoop cluster
• Stresses: TCO, scale-out, transform/big-query performance, ETL functionality

With the explosion of machine-generated data has come the need for a place to put it all, sometimes called the big bit bucket. This is like the investigative data mart for big databases, but more poly-structured. In some cases it is focused on data staging and transformation; but it can also be used for analysis in place.

The list of candidate technologies to run your bit bucket starts with Hadoop and Splunk.

Archival data store

• Kinds of data likely to be included: Operational, CDR (call detail record), security log
• Likely use styles: Archival, reporting (for compliance), possibly also investigative
• Examples: Any long-term detailed historical store
• Stresses: TCO, compression, scale-out, performance (if multi-use)

Analytic DBMS vendors have been insulting each other with the claim “that’s just an archival data store,” dating back at least to the first time Greenplum was deployed on an underpowered Sun Thumper system. Perhaps only Rainstor truly embraces the archival positioning, and I’ve become pretty dubious about their technical claims and their company alike.

Still, there’s a legitimate need for data stores — especially relational analytic DBMS that:

• Store data cheaply, with high rates of compression.
• Have decent performance if you do want to query the data.
• May have archiving/compliance-specific features as well.

Along with Rainstor, SAND and SenSage have at least partially targeted that use case. In addition, appliance vendors such as Teradata and Netezza try to have an archive-oriented product version in their lineups.

Outsourced data mart

• Kinds of data likely to be included: All
• Likely use styles: Traditional BI, investigative analytics, staging/ETL
• Examples: Advertising tracking, SaaS CRM
• Stresses: Performance, TCO, reliability, concurrency

Much of what happens in analytic database management can also be outsourced. Some applications that run via SaaS (Software as a Service) are analytic. I’ve had three different clients whose main business is picking marketing targets in various vertical segments; others who wanted to add analytics to what were historically OLTP applications; and others yet who just offered online business intelligence. Also, if your fundamental business is gathering data and reselling it to a variety of user organizations, that’s an analytic data management challenge. The possibilities expand from there.

Data outsourcers are in the IT business, and so their IT development is — hopefully! — more serious and less politically encumbered than at many conventional enterprises. Thus, legacy systems and master data management issues are commonly less prevalent, or at least more aggressively disposed of. The same, up to a point, goes for vendor politics.*  Multitenancy is commonly an issue, as is running in the cloud.

*Even so, there’s often That Guy who doesn’t want to migrate away from Oracle, no matter what.

Vertica gets the nod in a number of these cases; it’s cloud-friendly, and often the problem is naturally columnar. Other columnar products can be good choices too, with added brownie points for Infobright if the shop is MySQL-oriented anyway. Running Netezza or other appliances makes sense mainly if you’re pretty sure you want to keep operating your own data centers, but some data outsourcers are just fine with that assumption.

Operational analytic(s) server

• Kinds of data likely to be included: Customer-centric, log, financial trade
• Likely use styles: Advanced operational analytics
• Examples:
  • Lower latency: Web or call-center personalization, anti-fraud
  • Higher latency: Customer profiling, Basel 3 risk analysis
• Stresses: Performance, reliability, analytic functionality, perhaps concurrency

Even with eight different choices, I need a “catch-all” category; this is it.

Suppose you want to do reasonably sophisticated analytics, then use the results in operations. This is the classical challenge in integrating short-request and analytic processing. There are multiple ways to tackle it, embodying different trade-offs in cost, convenience, or analytic accuracy. If the platform on which you want to run your investigative analytics also has the reliability and concurrency appropriate for mission-critical operations, you’re set. Otherwise, you may want to pipe derived data into a more “industrial-strength” DBMS, ideally the one that runs your operational apps anyway

Another option is to integrate a limited amount of analytics immediately into your short-request processing system. For example, as bad as they are at the kinds of queries that require joins, NoSQL systems are often fast at simple aggregations. As MapReduce/NoSQL integrations mature, that option may not require pumping the data anywhere else for deeper analytics; even if it does, at least you’re starting out with the data in a convenient bit bucket.

Streaming/CEP-centric architectures could come into play as well. And it goes on from there. The possibilities in this last category are just too varied to generalize about.

So did I get them all? Or are there yet other analytic data management use cases that I don’t fit into my eight categories?

• Date arithmetic. This of course has been around for a very long — er, for a very long time.
• Time-series-aware compression. This has been around for quite a while too.
• “Time travel”/snapshotting — preserving the state of the database at previous points in time. This is a matter of exposing (and not throwing away) the information you capture via MVCC (Multi-Version Concurrency Control) and/or append-only updates (as opposed to update-in-place). Those update strategies are increasingly popular for pretty much anything except update-intensive OLTP (OnLine Transaction Processing) DBMS, so time-travel/snapshotting is an achievable feature for most vendors.
• Bitemporal data access. This occurs when a fact has both a transaction timestamp and a separate validity duration. A Wikipedia article seems to cover the subject pretty well, and I touched on Teradata’s bitemporal plans back in 2009.
• Time series SQL extensions. Vertica explained its version of these to me a few days ago. I imagine Sybase IQ and other serious financial-trading market players have similar features.

In essence, the point of time series/event series SQL functionality is to do SQL against incomplete, imprecise, or derived data.* For example, suppose in one time series events happen at times 3.00, 3.01, 3.03, and 3.05; in another time series events happen at times 3.00, 3.02, 3.03, 3.04, and 3.05; and you want to join the time series together. Then you can do an event series join — i.e., you can join on each of the times 3.00, 3.01, 3.02, 3.03, 3.04, and 3.05, using interpolated values to check WHERE conditions. Vertica says that the only interpolation methods anybody ever wants are “first value in the interval,” “last value in the interval,” and “linear average of the endpoint values” (I forget whether that’s weighted by time-distance from the endpoints, or is a simple arithmetic mean).

*This is a limited counterexample to my dictum that you should explicitly store derived data because it’s too much trouble to keep re-deriving it on the fly.

Also cool is the “CONDITION_TRUE_EVENT” syntax Vertica has had since Release 4.0, which generalized SQL 99 windowing; you now can look at all the rows that meet a specific criterion — via an arbitrary expression — rather than just being restricted to a row count or strict time duration. Vertica says it’s gone further in the direction of event series pattern matching in Vertica 5.0; I didn’t grasp the details, but it sounded philosophically akin to Aster Data’s nPath, albeit without the arbitrary-language procedural extensibility.

Finally, Vertica also gave me an imprecise-SQL example that has little to do with time series or other even series. Vertica has a concept of “range join,” implemented so that telecom firms can save space by storing partial IP addresses. I’ve noted before that while we should retain all human-generated data, it will never be practical to retain all machine-generated data (because its volume will keep going up based on the same technological factors that keep storage cost per unit volume going down). This sounds like one interesting (if specialized) approach to storing machine-generated data in summarized form.

Sybase said that Sybase IQ had > 2000 direct customers and >500 indirect customers (i.e., end customers of OEMs). That’s counting by customers; I know from prior discussions that Sybase IQ is running at close to two installations per customer. I also believe that Sybase counts different divisions of the same large enterprise as separate customers.

Vertica cited a figure of 500 customers as of April (end Q1?), which is close to 600 now, about 40% or a little more direct. The difference between this and a 2010 year-end figure of 328 is not only new sales, but also slow reporting by OEMs.  One cool figure — a single OEM reported 82 end sales in a single (quarterly?) report. And a number of those direct customers are substantial; Vertica’s customer logo page features lots of telcos, lots of internet companies, and the national operation of Blue Cross/Blue Shield.

Pay no attention to small inconsistencies in the number of Vertica direct customers (250 at year-end, no more than that now); Colin Mahony just estimates these numbers for me from memory, and minor inaccuracies are quite excusable.

Even cooler — Vertica reports 7 customers with a petabyte or more of user data each. About 5 of the 7 are obvious-suspect big-name firms; but unsurprisingly, those big names are NDA. I did secure permission to say that there are 2 telecom companies, a mobile gaming vendor, another internet company, and 3 financial services outfits of various kinds.

SAND Technology reported >600 total customers, including >100 direct. Since SAND has been around since the 1990s, those aren’t great average annual figures, but they’re probably more than many people (including me) thought.

Infobright reported around 200 total paying customers, 130 direct. There are surely a lot more users of open source Infobright, but precise numbers are of course hard to come by.

If I asked ParAccel in the April go-round, I’ve misplaced their answer, but back in October the figure was >30 customers, 2 of them over 100 terabytes. I’ve seen published figures of 40+ for ParAccel since.

• Web companies, many of which are already MySQL users.
• Telecommunication and similar log data, especially in OEM relationships.
• Trading/financial services, especially at mid-tier companies.

Key aspects of Infobright 4.0 include: 

• “Rough Query,” which lets you get approximate query results >10X faster than you could get precise ones, which is a good thing for iterative investigative analytics.
• The start of a plan — “DomainExpert” — to compress and otherwise optimize data in specific, commonly machine-generated patterns, such as URLs or CDRs (call detail records).
• “Distributed Load Manager” — i.e., load nodes that are separate from (and more parallelized than) query nodes.
• A Hadoop connector.
• Lots of cleanup and Bottleneck Whack-A-Mole, although I haven’t paid close attention as to which parts of that are truly new, and which were already handled in recent Infobright point releases.

Items on that list focused on the machine-generated data market include:

• DomainExpert — obviously.
• The Hadoop connector — also obviously.
• The Distributed Load Manager — why would you need such load speeds unless the data is flowing in from machines?

To understand Infobright Rough Query, recall the essence of Infobright’s architecture:

Infobright’s core technical idea is to chop columns of data into 64K chunks, called data packs, and then store concise information about what’s in the packs. The more basic information is stored in data pack nodes,* one per data pack. If you’re familiar with Netezza zone maps, data pack nodes sound like zone maps on steroids. They store maximum values, minimum values, and (where meaningful) aggregates, and also encode information as to which intervals between the min and max values do or don’t contain actual data values.

I.e., a concise, imprecise representation of the database is always kept in RAM, in something Infobright calls the “Knowledge Grid.” Rough Query estimates query results based solely on the information in the Knowledge Grid — i.e., Rough Query always executes against information that’s already in RAM.

To me, Rough Query is the most impressive part of the Infobright 4.0 announcement. DomainExpert sounds like it will be somewhat better than straightforward prefix/suffix compression, but Infobright hasn’t yet convinced me that the difference is substantial. Distributed Load Manager is indeed important, but only because Infobright doesn’t have a shared-nothing MPP (Massively Parallel Processing) option at this time. And the rest is mainly catch-up toward Infobright’s larger and more expensive peers.

• Consumer financial services. The idea here is to combine information about customers’ use of all kinds of services — banking, credit cards, loans, etc. SAS believes this is both for marketing and risk analysis purposes.
• Insurance. We didn’t go into detail.
• Mobile communications. SAS’ customers aren’t giving it details, but they’re excited about geocoding/geospatial data.

Meanwhile, in another interview I heard about, SAS emphasized retailers. Indeed, that’s what spawned my recent post about logistic regression.

The mobile communications one is a bit scary. Your cell phone — and hence your cellular company — know where you are, pretty much from moment to moment. Even without advanced analytic technology applied to it, that’s a pretty direct privacy threat. Throw in some analytics, and your cell company might know, for example, who you hang out with (in person), where you shop, and how those things predict your future behavior. And so the government — or just your employer — might know those things too.

Most notably, DataStax is introducing a software offering, whose full name is DataStax OpsCenter for Apache Cassandra. DataStax OpsCenter for Apache Cassandra seems to be, in essence, a monitoring tool for Cassandra clusters, with a bit of capacity planning bundled in. (If there are any outright operations parts to DataStax OpsCenter, they got overlooked in our conversation.)* OpsCenter has been in beta at a few places, with another beta version rolled out recently.

*Yeah, DataStax OpsCenter Release 1 sounds pretty boring. But it’s apt to be useful even so. And cooler stuff should come down the pike later on.

There will of course be a free-download version of DataStax OpsCenter, entirely uncrippled; you’re just not allowed  to use free-download DataStax OpsCenter with production applications. Production users of DataStax OpsCenter will need subscriptions. Much like Cloudera, DataStax is bundling product and support subscriptions, so that you can’t buy one without the other. The current Gold/Silver/Bronze trichotomy will be slimmed down to Mission-Critical/Premier, and you’ll be allowed to have different levels for different application clusters.

Finally, a few customer notes:

• DataStax has >50 subscription support customers.
• One DataStax customer has 400 Cassandra nodes.
• DataStax’s major industry segments are web (of course), government/intelligence, and telecom.
• Separately – I’m not sure why this is separate – DataStax thinks the next market it will penetrate is real-time analytics. That means online counts or other aggregations, although presumably not at a Skytide level of sophistication.
• Netflix has nice things to say about HBase and Cassandra alike.

There are two technology trends driving electronic privacy threats. Taken together, these trends raise scenarios such as the following:

• Your web surfing behavior indicates you’re a sports car buff, and you further like to look at pictures of scantily-clad young women. A number of your Facebook friends are single women. As a result, you’re deemed a risk to have a mid-life crisis and divorce your wife, thus increasing the interest rate you have to pay when refinancing your house.
• Your cell phone GPS indicates that you drive everywhere, instead of walking. There is no evidence of you pursuing fitness activities, but forum posting activity suggests you’re highly interested in several TV series. Your credit card bills show that your taste in restaurant food tends to the fatty. Your online photos make you look fairly obese, and a couple have ashtrays in them. As a result, you’re judged a high risk of heart attack, and your medical insurance rates are jacked up accordingly.
• You did actually have that mid-life crisis and get divorced. At the child-custody hearing, your ex-spouse’s lawyer quotes a study showing that football-loving upper income Republicans are 27% more likely to beat their children than yoga-class-attending moderate Democrats, and the probability goes up another 8% if they ever bought a jersey featuring a defensive lineman. What’s more, several of the more influential people in your network of friends also fit angry-male patterns, taking the probability of abuse up another 13%. Because of the sound statistics behind such analyses, the judge listens.

Not all these stories are quite possible today, but they aren’t far off either.

One of the supporting trends, pretty obvious, is that there is a lot more electronic information than there used to be. Indeed:

• Sufficient information exists to provided a very detailed picture of our activities.
• Much of it is recorded for very good and beneficial reasons. We wouldn’t want that  part to stop.
• This information is inevitably available to government.

Here’s what I mean by the inevitability claim. Whether or not you think anti-terrorism concerns are overblown, as a practical matter your fellow voters* will allow a broad range of governmental information access. Besides, just the widely-available credit card and similar commercial data is enough to provide a fairly detailed picture of what you’re up to. In most countries, anti-pornography, anti-file-sharing, and/or general civilian law enforcement efforts serve to strengthen the point further.

*If you live in a country too unfree for voters to much matter, then it is surely also the case that governmental information has few practical limits.

Examples of information being tracked (more particulars were covered in the first post of this series):

• Almost everything we buy is recorded, via credit card transactions, point-of-sale data, and/or website transaction records. This data is summarized in files covering 100s of millions of individuals, with 1000s of fields per person. Those files can be used for a broad variety of business or law enforcement purposes.
• That data gives a great picture of what we eat, where we commute or travel, what we pay attention to, and so on.
• All our other financial information also passes through computer systems, such as at banks.
• Increasingly, our physical movements are tracked more directly, via cell phones (our own), police cameras, and the like.
• Other than face-to-face conversations, almost all our communications are electronic. Even social media non-adopters rely heavily on telephones, email, and the like.
• Increasingly, our reading and viewing entertainment choices are electronically recorded as well.

Most of that data is available to law enforcement departments. Much of it is available to commercial companies as well.

And these vast amounts of data will hardly go to waste. The second major technological trend in play is that the data can be much more effectively analyzed than before. New kinds of or effectiveness in analytic profiling create whole new levels of exposure (using the word “exposure” in its most literal sense), in at least three ways:

• Relationship profiling.* Relationship analytics technology has been around for a while. When it’s used to find bad guys (terrorists, fraudsters, etc.), that’s one thing. But some of the marketing uses are spookier. Marketing-like uses applied back to governmental surveillance could be spookier yet.
• Propensity profiling.* A huge fraction of what happens in big data analytics is figuring out what you’re likely to buy, vote for, look at, click on, react to, or think. Marketers getting that right can be a bit creepy. So can marketers getting it wrong. Governments doing the same thing could be much creepier yet.
• De-anonymization.* You may think you can be anonymous online, but you really can’t. Also, it’s getting ever harder to keep your roles or activities online separate from each other.

* I just coined the terms “relationship profiling” and “propensity profiling.” “De-anonymization,” however, has been in use for a while.

Classical relationship profiling questions include assessing who has a close relationship with whom, who influences whom, who influences lots of people, etc. The most obvious data to infer this from is communication — who called whom, how long they talked, who they called next, what time of day this all happened, and so on. Anti-terrorist uses are obvious. A major marketing use is telcos — who of course have this data — deciding who to offer their best deals to, by trying to identify who influences the most other customers. These calculations of course involve comparing lots of data, mainly about people who are NOT targets of terrorist investigation or preferential telephone service pricing.

Much of Facebook’s $50 billion valuation hinges on the assumption it can do similar things based on the “social graph” it infers from informal communication among friends. To date that assumption has been questionable, but we’re still in the very early days. Meanwhile, cruder methods of analyzing social influence are used. But the trend is clear — marketers want to use technology to identify social leaders, influence them however they can, and hope that the rest of us follow along baaing. Up to a point, that’s actually OK — learning things from our friends and acquaintances is an important and pleasant part of living in a society. And political campaigners have been doing it for generations, in the most low-tech of fashions. Still, it’s one thing for such targeting of leaders to be transparent; if done surreptitiously, it suddenly starts to feel a lot more sinister.

For years, propensity profiling has been an area of huge investment and technological progress. It’s the central application of big data analytics, and the heart of the business for many companies I write about, or that are my clients. Credit files, web logs, other marketing responses, census information, and other data are combined to infer:

• Your income, household composition, age, race, education, and other basic demographics.
• Your buying, voting, reading, viewing, and other consuming interests in minute psychographic detail.
• Your feelings about particular companies and brands, your propensity to become or stop being their customer, and what kinds of advertisements or offers it would take to influence you.
• Your status as a credit risk.
• The chance you are committing or will commit fraud.

This has been going on since at least the 1990s, especially in service industries with “loyalty card” kinds of programs, such as retail or travel/leisure. In the credit case it’s been going on longer than that. But new data sources, processed by new analytic technologies, have brought the practice to a vastly greater height.

Finally — in case you care about being anonymous online, you’re running out of luck. De-anonymization analytics are getting too good. The Electronic Freedom Foundation’s de-anonymization overview in 2009 was one of many articles pointing out that it often was possible to attach a specific name to online activities that in theory don’t track personally identifiable information. Meanwhile, at a talk I attended in May, 2010, comScore spoke of its successful efforts to tie various anonymous online activities, such as visits to different websites, to each other. And after I entered “usinger.com” into my browser address bar, I started seeing ads for Usinger sausages at a variety of prominent websites.

I’m not sure how much of a privacy threat de-anonymization technology is in and of itself, but it certainly provides support to both relationship and propensity profiling.