• People
• Financial trades
• Electronic networks
• Everything else

The most varied, interesting, and valuable of those four categories is the first one.

That may change some day, with the growing importance of machine-generated data, and of big-data science in particular. But I think it’s a fair assessment at the present, and for at least the next few years.

Some of the most interesting use cases are concentrated in the areas of identifying individuals, groups of people, or behaviors of (groups of) people. For example:

• comScore works hard to identify individual web surfers – i.e. to deanonymize them — even though they may have given incomplete or false personal information.
• Other companies at least try to figure out which information in a user’s profile is unreliable, so as to classify them better. (Yes, there are 62-year-old video-game-obsessed Lady Gaga fans, but that’s generally not the way to bet.)
• Multiple telecom vendors try to identify who their most influential customers are (to a first approximation, they’re the ones most often called by the most people, but it surely gets more sophisticated than that). This information is then used to reduce churn, either by working hard to retain those users, or – if they do churn – to move very fast to retain the business from their friends.
• Other kinds of companies do similar kinds of analysis, to the extent that they have enough of a social graph to do so. (This application is a case where the term “social graph” is not a misnomer.)
• Turing detectives (I just coined that phrase) try to determine whether users are humans or bots.
• Central to detecting insurance fraud is identifying suspiciously close connections between claimants, service providers, and so on.
• Identifying groups of people is also important in flagging insider trading. Even more important are other kinds of analysis, along the lines of “is this normal innocent trading behavior?”
• Intelligence agencies try to detect networks of terrorists and their sympathizers. They further try to identify unusual patterns of communication or meetings along those networks that might indicate terrorist acts are being planned. (Civilian law enforcement agencies can use similar techniques.)

In most cases, the analysis and/or run-time execution of the relevant models is done with the help of analytic DBMS. Other technologies that come into play include non-DBMS MapReduce (Hadoop), graph engines, and CEP (Complex Event Processing). The vendor most heavily represented on that list is probably Aster Data, because:

• Aster Data is focused on hard-core analytics.
• I talk a lot with Aster Data, and in particular had a long, detailed use-cases discussion with them last week.
• The comScore example happens to come from a speaker at an Aster event I also participated in.

And by the way, all this only scratches the surface of what will be possible down the road. It’s based mainly on where you live, what you purchase, how you behave on websites, and who you communicate with. Other kinds of data, which could be used to be yet more intrusive, generally aren’t involved.

I actually have two points in drawing up this list. One is golly-gee-whiz about how a lot of analytically sophisticated applications are actually getting into production. The other is to highlight the privacy and liberty threats If This Goes On Unchecked (which is why I didn’t include some other less-people-focused examples). There’s also a related danger that, to the extent we don’t get some smart regulations to keep us safe(r), we’ll get a bunch of stupid regulations instead.

The Analytic Era has only just begun.

• Comes in large volumes
• Is machine-generated
• Is augmented by synthetic and/or derived data
• Has a spatial and/or temporal structure

In those respects, it is akin to some of the hottest areas for big data analytics, including:

• Investment trade data – big, partly machine generated, augmented (often), temporal
• Web/network log data – big, machine-generated, post-processed into derived form, temporal
• Marketing analytic data – big, post-processed into derived form
• Genomic data

So when Jacek Becla started the XLDB conferences on the premise that scientific and big data analytic challenges have a lot in common, he had a point. There are several tough database problems that the science-focused folks have taken the leading in thinking about, but which are soon going to matter to the commercial world as well. And that’s one of two big reasons why you should consider participating in XLDB4, October 6-7, at the SLAC facility in Menlo Park, CA, as an attendee, sponsor, or both.

The other big reason is that it is important for the world that XLDB succeed. Computer technology to analyze global warming is lacking; better database technology is one of the ways it could improve. Database technology also has important potential contributions to make in medical research and other worthy endeavors, and in a lot of purer science too (Jacek himself is an astronomy guy).

Other reasons to get involved with XLDB4 include:

• It doesn’t cost much. The whole thing is done in academic conference dollar amounts, with low attendance fees, the venue use probably donated by SLAC, and (I would guess) low sponsorship fees as well.
• It’s fun. I think I attended more sessions in two days at XLDB3 than at the all the other conferences I went to last year combined.

All in all, I intend to spend three whole days attending conference sessions that week, which is something I almost never do. But it’s for my two favorite causes – scientific data management and liberty/privacy. Stayed tuned for news on the latter front soon.

Related links

• The XLDB4 conference web site
• My original post on issues in scientific management, most of which will be discussed at XLDB4

• If you are or want to be a serious MapReduce user – and you’re past the “play around over the weekend” stage — you probably should have either:
  • A serious non-DBMS MapReduce distribution.
  • MapReduce integrated into your analytic DBMS.
  • Both.
• The obvious choice for non-DBMS MapReduce is Hadoop.
• The obvious choice for a Hadoop distribution is Cloudera Enterprise.
• Cloudera Enterprise has three main aspects, in an inseparable bundle:
  • Distributions for a double-digit number of open source projects. It’s nice having all that in one package – unless, of course, you like playing with Tinkertoys.
  • Proprietary Cloudera code.
  • Cloudera support.
• Cloudera says its proprietary code is and in the future is planned to be concentrated – at least in large part — on integrating open source technology with closed source products. This has the virtue of being targeted directly at that segment of the market which has proven it’s actually willing to pay money for software.
• Cloudera Enterprise areas of focus, now and in the presumed future, include:
  • Core Hadoop engine, which Cloudera says is quite predictably and appropriately evolving more slowly than the tools around it.
  • Development, management and administrative tools, including:
    • Pig and Hive. Cloudera says >70% of Facebook Hadoop jobs are initiated through Hive, and the same is true of Yahoo and Pig.
    • Connectivity to commercial tools.
    • The product formerly known as “Cloudera Desktop.”
  • Workflow, which in this context refers to letting you create a Hadoop application as a sequence of small steps, rather than forcing you to kluge it into being one unwieldy thing. At the moment, this is much less widely adopted than Pig and Hive, but Cloudera has high hopes for it, because of its obvious benefits in modularity and manageability.
  • Quasi-DBMS technology. Besides Hive and Pig, this includes HBase. Cloudera says there has been considerable demand for HBase, and it is pleased that project is now mature enough to ship. Cloudera stresses that it intends HBase not for OLTP, but as an adjunct to analytic processing. E.g., Cloudera suggests HBase would be a fine vehicle for replicating dimension tables across each node of a cluster.
  • Data connectivity, e.g. to MySQL or to sensor log files.
• Cloudera Enterprise pricing is well below DBMS prices – not by a full order of magnitude, if I’m right about everybody’s quantity discount policies, but even so by a lot. Details are NDA.

Cloudera sometimes sends confusing signals about its beliefs and strategies. For example, one can get different stories depending on whether one talks to:

• Somebody at Cloudera who comes primarily from the user and open source communities.
• Somebody at Cloudera who has actually worked at a software company before.

But I predict that Cloudera will now stick for a while with more or less the strategy outlined above.

Naturally, we also talked about Hadoop adoption. Highlights of that part – no doubt somewhat biased towards Cloudera’s own customer base — included:

• Notwithstanding eBay’s prior skepticism about MapReduce, it is quoted saying nice things in a Cloudera press release, and has apparently become quite a large Hadoop user, starting out with a search-quality use case.
• Typical Hadoop deployment sizes are 10 nodes or so when experimenting, 80-500+ in production.
• 10 terabytes/node – I’m pretty sure Cloudera meant of user data — is not inconceivable, so a cost-conscious 500-node user could have 5 petabytes of data managed by Hadoop.
• Cloudera has half a dozen customers at the 75+ node production level.
• Web and financial services are the two vertical markets moving most aggressively into Hadoop production. The government is also in significant Hadoop production, but the details of that are classified.
• Web uses for Hadoop include:
  • Clickstream – sessionization, etc. – that’s a super-mainstream use.
  • Search – analyzing search attempts in conjunction with structured data.
  • Machine learning (for ad serving, etc.).
• Financial services uses for Hadoop include:
  • Internal trading rule enforcement/fraud detection.
  • Complex ETL.
  • Portfolio risk assessment (typically overnight).

None of this is inconsistent with previous surveys of Hadoop use cases.

Various users talked at the Hadoop Summit this week. I wasn’t there, and won’t write about their stories for now. That said, Twitter’s slide deck from same has some interesting stuff, including:

• 7 TB/day ETLed from MySQL.
• Petabytes-being-stored accordingly coming soon.
• Open sourcing their ETL tool Crane.
• 3-4X LZO compression at little CPU cost.
• HBase is a more usable for them than HDFS, which isn’t mutable enough.
• Pig = 5% of code and coding effort vs. vanilla Hadoop at 30% or less performance hit.

There’s something akin to “social graphs” and “social network analysis” that is more or less worthy of all the current hype – but graphs and network analysis are only a minor part of the whole story.

In particular, the most important parts of the Facebook “social graph” are neither social nor a graph. Rather, what’s really important is an aggregate Profile of Revealed Preferences, of which person-to-person connections or other things best modeled by a graph play only a small part.

Let me hasten to note that – even when viewed narrowly — the ideas of “social graph”and “social network analysis” do have significance. Nontrivial use cases to date for big data social network analysis include:

• Intelligence agencies identify and analyze terrorist networks. Corporations and civilian law enforcement do the same for fraud networks.
• Telephone companies use calling data to figure out which of their customers are most likely to influence which other customers in the decision to keep or change service providers. (Frankly, I find that rather creepy.)
• Social networks figure out which other members you’re likely to know, and encourage you to connect with them.

Epidemiologists aspire to add to that list, based on their success to date using much more micro forms of social network analysis. But after that, I’m running out of examples. Sure, graph analytics is good for a bunch of other things (e.g., biology at the genetic or molecular level), but those have little or nothing to do with “social graphs” or social network analysis as they are commonly understood.

Note: Of course, it is also the case that everything can be modeled by entity-attribute-value triples, and those can always be modeled by graphs. But so what?

Let’s consider what, in a marketer’s ideal world, would go into your Profile of Revealed Preferences. Raw data might include:

• Personally identifyING information. Duh. This is what makes everything else possible.
• Purchase transaction data. Lots of it. Like, everything on your credit card statements.
• Demographic and lifestyle information. Address, date of birth, educational history, race, household composition, and so on.
• Affiliations. Politics, religion, group membership of any kind. (OK, that’s partly social.)
• Explicitly stated opinions, preferences and desires, including:
  • Any surveys you have filled out.
  • Any recommendations you have made (e.g., through the Facebook Like feature).
  • The text of anything you’ve written and posted – and, very ideally, of your private emails as well.
  • Any wish lists you’ve filled in.
• Attention information. What you clicked on, what you looked at, and all that stuff website owners track.
• Your movements, to the extent they are tracked. (E.g., via Foursquare and the like.)
• Your gaming activities and the like. (This is social mainly to the extent it overlaps with other categories I’ve already mentioned.)
• Your medical information.
• Who you communicate with, and what you communicate with them about. (Hey! There’s something else social!)
• Similar information about the people you communicate with.

My core privacy thoughts about that data include:

• Individuals deserve the right to control all that information. At a minimum, they deserve total control over how the data (raw or derived) is passed from the service – e.g., website – where it naturally resides (e.g., where it is originated) to any other place.
• Given a chance, individuals would make fine-grained choices about what parts of their Profile of Revealed Preferences are available to which organizations. Reasons include:
• Individuals have rather complex trust relationships with different kinds of merchants and marketers.
• Consumers get different benefits from sharing information with different kinds of merchants and marketers. (Sometimes personalization is a large benefit. Sometimes it’s just creepy. And some companies actively bribe you to give them information they can use to sell to you.)

When one frame things this way, two rather difficult technological questions naturally arise.

1. Suppose, implausibly, that a single entity were allowed to control and use (for marketing) all of your Profile of Revealed Preferences information. How would they store and analyze it?
2. How does the answer to #1 change because control over the information will, in fact, be fragmented?

It’s tough enough to answer these questions for data about one person. Trying to include all but the simplest information about other people is and will for years remain quite infeasible. So, for the most part, this is not “social” information.

It’s also not naturally a “graph.” Similarly, it is not a good candidate for network analysis. To see why, let me outline why I used the name “Profile of Revealed Preferences”:

• The reason marketers want this data is, mainly, because they want to know what appeals to you, and how strongly you feel about it.
• The analytic process often entails taking explicit choices you have made, and inferring other preferences from them.
• The output of the analytic process is often one or more “scores” that then get plugged into various selection algorithms to determine what you should be shown or offered. At least implicitly, these algorithms are predicting what you will or won’t respond well to.

Not much graph-like there.

This post has gotten pretty long, so I’ll stop here without spelling anything else out. But questions I still hope to address down the road include:

• How should Profile of Revealed Preferences data be stored?
• Suppose we want to pass around derived results and not the raw data. How could we ever get to standards that would make such interchange realistic?
• If we only have raw data to pass around, what are the implications for privacy, liberty, and the structure of the online industries?

The main new activity I know of has been in the open source SciDB project.  

• A company called Zetics has been started to commercialize SciDB. As of now, the entire staff seems to be CEO Marilyn Matz, techie Paul Brown, and part of Mike Stonebraker. Marilyn says Zetics has some venture capital, but even under NDA didn’t tell me who it was from. Zetics does not have its own web site.
• Marilyn tells me there are 20-25 contributors to SciDB, led by Paul Brown and Mike Stonebraker. Brown is full-time. Persistent Systems has been donating the efforts of a few of its employees. Some LSST folks have been doing SciDB work backed by grant money. Most or all of the rest seem to be purer volunteers. Some Russians have been particularly active.
• Release 0.5 of SciDB is expected in June. Release 1.0 is expected in September. This is a rewrite; prior demo code has been scrapped. Perhaps not coincidentally, it’s also a small slip from prior project plans.
• The array data model is an example of what’s being implemented first. (Duh — you can’t have a DBMS without a data model.) Support for uncertainty is an example of what’s been deferred until later.
• As has been clear since XLDB3 last August, one major target market for SciDB is genomic research.
• It’s obvious that the oil and gas industry, with all its geospatial data, should be interested in SciDB. But there’s not much activity in that regard; outreach is evidently needed. If you can think of somebody in that sector (or anywhere else) who should be alerted to SciDB, please ping them.
• Interest from web analytics users in SciDB seems to have receded a bit from the days when eBay almost funded the project.

In other scientific data management news,

• Microsoft put out a book called The Fourth Paradigm on scientific database management. The whole thing can be downloaded, very officially, as a giant PDF. I think it’s worth skimming. I don’t think it’s worth actually reading. (I did read it.)
• XLDB4 will be at Stanford October 5-7. Unlike prior XLDBs, it will have an open (i.e., no invitation required) part.

Finally, you surely are aware of the whole “Climategate” mess, in which major climate researchers’ email was hacked and many unkind conclusions were drawn. Well, one of the most technical parts of the disclosure was in a long series of Read Me files, in which an unfortunate programmer lamented about the difficulty of reconstructing published results from files at hand. These turned out to illustrate a classic problem that SciDB or alternatives are meant to solve:

• Raw data was impossible to use without various adjustments to regularize it (the word “regridding” comes up a lot, for example). Massaging was needed before analytics could be done on it.
• The raw data was thrown out or lost, and could not be reconstructed (why they couldn’t have asked the suppliers of the data to give it to them again was unclear in this case, since it wasn’t original experimental data).
• It was thus impossible to massage the data in any new or improved way.

Meanwhile, Truviso — which never got much market traction in the first place — was in hiding; Roman Bukary never did keep his promise to brief me on the company’s new and improved strategy. Then Truviso had yet another management change, amidst rumors that it was repositioning away from CEP. As per a press release Truviso emailed today, that’s now official, with Truviso’s main business being something to do with web analytics.

• Vertica plans to build a massive, world-conquering sales force.
• If Vertica dips back into negative cash flow to do that and has to raise more venture capital, so be it.
• “Triple-digit” revenue growth is expected for this year.

*I’ve since heard more both from Ralph and his former colleagues, and I’m comfortable taking the move more or less at face value — for some reasons he doesn’t want to spell out, Ralph really wanted to move back home to South Africa.

While they were at it, Vertica also put out a press release reporting very good success in the social gaming market. The biggest and best known of the bunch is Zynga. Three months ago, Wayne Eckerson had figures of 3 TB/day added to the database, 200 nodes, and >40 million users. Now Zynga is using a figure of >65 million daily users and 230 nodes. More precisely, at Zynga:

• There are two Vertica databases with identical data.
• Each Zynga Vertica database runs on 115 nodes.
• Zynga’s two Vertica database clusters are used for different applications.
• It’s undisclosed exactly what Zynga runs on what Vertica cluster. But best practice would be to put mission-critical, fast-response stuff on one cluster, and use the other for longer-running or less-critical queries — plus have it be available as hot standby — given that I don’t see much reason to put data geographically close to users around the world for reasons of latency or whatever.
• An undisclosed amount of data, amounting to all of what Wayne earlier estimated at 3 TB, is added to each of Zynga’s Vertica databases daily.

In other news, Vertica now states its customer count as being >130.

• Computer, network, and other equipment logs
• Satellite and similar telemetry (whether for espionage or science)
• Location data such as RFID chip readings, GPS system output, etc.
• Temperature and other environmental sensor readings
• Sensor readings from factories, pipelines, etc.
• Output from many kinds of medical device, in hospitals and (increasingly) homes alike

The core idea here is that human-generated data can grow only as fast as human data-generating activities allow it to, but machine-generated data is limited only by capital budgets and Moore’s Law.  So machines’ ability to generate data is growing a lot faster than humans’.

Up to this point, I think there’s broad agreement, at least on the part of anybody who’s thought about it this way for very long. But that still leaves open questions as to which kinds of machine-generated data will matter first. The big five that matter right now are:

• Web logs (partially machine-generated, but tied to human actions)
• Call detail records (CDRs — ditto)
• Financial instrument trades (some purely machine-generated, some human-based)
• Network event logs (commonly associated with web logs)
• Telemetry collected by the government (especially for intelligence purposes)

A large fraction of all the 100 TB+ or petabyte+ data warehouse activity I know of falls into those areas.

Following along quickly are:

• Online game data. Since late last year, online game companies have come up over and over again as an important category of data warehousing/analytics users. Like most of the categories above, the gaming area actually features a hybrid between human- and machine-generated data.
• Genetic research data, although I don’t know to what extent the investment in data gathering is concentrated among the few obvious big pharmaceutical companies. Other health care data (research or clinical) will come along too, but doesn’t seem to be there yet.

Until recently I would have added:

• Energy exploration, energy production, energy refining, and/or utility network data

But while those areas seemed poised to get hot last year, I haven’t heard much about them the past few months, with a few exceptions:

• Accenture’s observation that new smart grids will generate up to eight orders of magnitude more data than old dumb grids do
• The recent article about the Terralliance fiasco (new kinds of oil exploration analytics, going beyond seismological data)
• Lots of  concern about security flaws in utility smart grids.

Finally, I’ve been assuming that a big area going forward is location data, especially personal movement data. The data volumes involved could be similar to or even greater than those of CDRs. But privacy concerns with that are obviously immense. (Of course, in the case of Foursquare, this sort of overlaps with freely-shared game data.)

If you want to make all this more tangible in your mind, one area to look for ideas is in the huge amount of news about various kinds of innovative sensors. Sources include:

• Somebody named Landon Cox, who maintains a couple feeds of sensor news.
• A Twitter feed, apparently associated with a Sensor Expo.
• Another Twitter feed, this one from Sun Labs. (I have no idea what Oracle is or isn’t doing with the Sun SPOT project that links to.)
• Yet another Twitter feed.

Nonetheless, OLTP/general-purpose data management startup activity has recently picked up, targeting what I see as some very real opportunities and needs. So as a jumping-off point for further writing, I thought it might be interesting to collect a few observations about the market in one place.  These include:

• Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.
• By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value.
• Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.
• It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.
• The era of silicon-centric relational DBMS is coming.
• The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds.
• Users’ instance on “free” could be a major problem for OLTP DBMS innovation.

I shall explain.

Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.

• OLTP applications are more complex than analytic ones, and hence more tightly wired into particular brands of DBMS. For example, third-party packaged OLTP applications are typically portable among only a few brands of DBMS. But third-party business intelligence tools, and the BI “applications” built in them, are more easily and widely portable.
• Specific technical observations such as “OLTP apps tend to use stored procedures, which are DBMS-specific” or “OLTP apps tend to have lots and lots of tables” serve to underscore the first point.
• An enterprise’s highest-value data is commonly the financial stuff handled by its core OLTP systems, so those are the last things they want to mess around with just to get some cost savings. Security, high availability, and so on are major considerations that can outweigh cost.

By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value. Indeed, “OLTP” is often a misnomer, which is why I tend to go with “general-purpose” or some similarly wishy-washy phrase instead.

• In theory, this is a ripe area for what I’ve called mid-range DBMS.
• The big brand vendors try hard to keep as many of those databases for themselves as they can. Enterprise-wide license pricing helps. Going forward, so will virtualization/consolidation strategies, such as Oracle’s Exadata-centric approach.
• A variety of mid-range DBMS alternatives beyond the big brands have technical merit, at least in some cases and configurations – MySQL, PostgreSQL, Intersystems Cache’, and so on.
• The only such mid-range DBMS alternative with much large enterprise business momentum, however, appears to be MySQL.

“General-purpose” might be a better term than “OLTP” anyway.

• I don’t have a link, but it’s widely agreed that over half of the processing on an “OLTP” enterprise app is commonly reporting and so on.
• “Operational BI” is progressing by fits and starts, but it is progressing.
• Anything customer-facing — web-based, call center, or otherwise — is likely to include a heavy dose of “real-time” analytic optimization.

Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.

• VoltDB is the main exception that jumps to mind.
• This isn’t true in the analytic DBMS area, where Netezza, Greenplum, Aster, Vertica and others started from PostgreSQL’s code, APIs, or both.

It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.

• MySQL is a limited product without all the third-party storage engines that are being developed.
• Oracle’s promise of MySQL good behavior has an expiration date.
• None of the MySQL front-end alternatives are remotely mature yet.

The era of silicon-centric relational DBMS is coming.

• I think “silicon” means “solid-state memory” as much as or more than it means “RAM,” but that’s not yet certain.
• What is pretty certain is that, thanks to Moore’s Law, some kind of silicon will increasingly replace disk.
• Oracle’s increasingly Flash-centric story is a challenge to everybody.
• RAM-centric VoltDB will launch fairly soon. (By the way, while VoltDB still has a lot in common with H-Store, they’re not exactly the same thing. And H-Store research is progressing too.)
• RethinkDB is being developed, focused directly on solid-state memory. Based on the sparse information available online, RethinkDB sounds somewhat like a dumbed-down H-Store.
• New disk-based vendors may never optimize their use of disk, instead targeting a solid-state future. (E.g., I think Akiban should and quite well might follow this path.)

The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds. We hear that from the NoSQL guys all the time. But I also just heard it from Akiban.

Users’ instance on “free” could be a major problem for OLTP DBMS innovation. Vendors of new OLTP data management technologies often feel obligated to open source their products, notwithstanding the historical lack of revenue in the open source OLTP DBMS market. As just one of many examples, Nova Spivack wrote:

I have recently seen some new graph data storage products that may provide the levels of scale and performance needed, but pricing has not been determined yet. In short, storage and retrieval of semantic graph datasets is a big unsolved challenge that is holding back the entire industry. We need federated database systems that can handle hundreds of billions to trillions of triples under high load conditions, in the cloud, on commodity hardware and open source software. Only then will it be affordable to make semantic applications and services at Web-scale.

I hear similar things from other startups, who evidently believe they need and/or are entitled to enjoy sophisticated, high-performance, zero-cost, specialized database management technology.

As costs go down, the wisdom of keeping detailed data goes up. I’d go so far as to say that every piece of data generated by a human being should be preserved and kept online, legal and privacy considerations permitting.* Most forms of capital-, labor-, and/or location-based competitive advantage being commoditized and/or globalized away. But information remains a unique corporate asset. Don’t discard it lightly.

*Unless there’s an explicit law mandating data destruction, legal considerations should permit. The idea “Let’s destroy something of irreplaceable value today, against the possibility we might be brought to judgment tomorrow” is both morally and pragmatically weird. Privacy, however, may be a different matter.

That applies to the structured/tabular kinds of data I tend to focus on in this blog. It applies even more to anything that’s like a document (or email, instant message, whatever) somebody has taken the trouble to place into words.  A top document-oriented archiving analyst (and my good friend), David Ferris, quite agrees. As David puts it:

I think we’ll end up archiving everything, except egregious garbage like spam:

• It’s too hard to get users to conform to policy.
• Automated methods of capturing a human-understandable policy, for example “tax records,” are too hard to implement through automatic filters. The filters are too inaccurate.
• It’s impractical to get users to classify everything, and automatic classification is too crude.
• You never know what you might want later. Stuff you think you won’t want now may end up being very useful.
• The cost of storage is trivial when looked at on a per-user basis.

In particular, I think information destruction is a crude instrument for the protection of privacy, wasteful at best, and likely to be vigorously resisted by governments and large businesses.  For example:

• Businesses are increasingly subject to retention-oriented compliance regulation. Your lawyers may want you to destroy information that could be used to sue you, but governments won’t let you.
• Information about individuals’ web surfing is being retained, under law, so that they may be fingered later for pornography consumption or illegal file sharing. I deplore some of the ways web-surfing data can be and is being used, and want laws passed to rein them in. But the retention will happen.
• Marketers want all that data. Duh.
• Electronic health records are coming — slowly, but they’ll get here some day.

Besides, archiving technologies are getting ever more cost-effective.