• More-of-the-same in line with MarkLogic’s core positioning.
• A new bi-directional Hadoop connector.
• A free MarkLogic Express edition, limited in license terms more than in actual features, as per Slide 27 of the deck MarkLogic graciously supplied for me to post.

Also, MarkLogic is early with a feature that most serious DBMS vendors will soon have – support for tiered storage, with writes going first to solid-state storage, then being flushed to disk via a caching-style algorithm.* And as befits a sometime search-engine-substitute, MarkLogic has finally licensed a large set of document filters, from an Australian company called Isys. Apparently, the special virtue of the Isys filters is that they’re good at extracting not only text, but metadata as well.

*If there’s a caching algorithm that doesn’t contain a major element of LRU (Least Recently Used), I don’t recall ever hearing about it.

MarkLogic seems to have settled on a positioning that, although distressingly buzzword-heavy, is at least partly based upon reality. The real part includes:

• MarkLogic is a serious, enterprise-class DBMS (see for example Slide 12 of the MarkLogic deck) …
• … which has been optimized from the getgo for poly-structured data.
• MarkLogic can and does scale out to handle large amounts of data.
• MarkLogic is a general-purpose DBMS, suitable for both short-request and analytic tasks.
• MarkLogic is particularly well suited for analyses with long chains of “progressive enhancement” (MarkLogic’s favorite term when talking about derived data).
• MarkLogic often plays the role of a content assembler and/or search engine, and the people who use MarkLogic in those ways are commonly doing things that can be described as research and analysis.

Based on that reality, MarkLogic talks a lot about Volume, Velocity, Variety, Big Data, unstructured data, semi-structured data, and big data analytics.

My November, 2010 overview of MarkLogic technology remains pretty relevant. One correction, however: Node heterogeneity configurations, in which “data” and “evaluation” nodes reside on separate servers, are the exception rather than the rule.

Like Vertica, MarkLogic has laudably said that true academic researchers can get MarkLogic for free without the severe license restrictions. Free MarkLogic should be of particular interest to researchers who:

• Are studying natural networks or graphs, such as social networks or biological pathways. (This might be a fit in the social or biological sciences.)
• Are managing metadata for, say, a variety of disparate kinds of experimental files. (This might be a fit anywhere in the natural sciences.)
• Are managing actual documents, images, videos, etc., or data about such things. (This might be a fit in the humanities or social sciences.)

MarkLogic provided some disclosable financial substance by email, which I shall quote verbatim:

• MarkLogic has 45% revenue growth and 55-60% license growth year over year.
• We expect to finish this year with over $85 million in revenue, up from $55 million last year.

Arithmetical purists might note that 85/55 is more than 145%, but I’m just going to settle for the information I got and move on.

Edit: I posted separately about the MarkLogic Hadoop connector. As for that Hadoop connector – stay tuned for a short follow-up post, as writing about it now would not be convenient. (My backup discipline isn’t what it should be, and the only copy of my notes about that product is on a heavy tower computer in a house that doesn’t have working power.)

• Academic scientists like their software to be open source, for reasons that include both free-like-speech and free-like-beer.
• What’s more, they like their software to be dead-simple to administer and use, since they often lack the dedicated human resources for anything else.

Even so, I think that academic researchers, in the natural and social sciences alike, commonly overlook the wealth of commercial software that could help them in their efforts.

I further think that the commercial software industry could do a better job of exposing its work to academics, where by “expose” I mean:

• Give your stuff to academics for free.
• Call their attention to your free offering.

Reasons to do so include:

• Public benefit. Scientific research is important.
• Training future customers. There’s huge academic/commercial crossover, especially as students join the for-profit workforce.

The biggest issue is probably large-scale database management. There’s a feeling, permeating for example parts of the XLDB conference and the associated SciDB project, that data stores suitable for holding large amounts of data are either:

• Hadoop or
• Forbiddingly expensive.

I think that’s overstated. In particular:

• You can put >10 terabytes of machine-generated data (or any other kind) into Infobright and have it well taken care of; Infobright is open source.
• You can put >1 petabyte into [name redacted],* among others; [name redacted]* should be out soon with a generously free offering for academic users. Edit: That would be Vertica.
• Conventional relational queries, graph analysis, statistical analysis preparation and more can all be much faster in a good analytic DBMS than in alternative kinds of data stores.
• Integration between SQL and other analytic languages is ever improving, as analytic DBMS evolve into “analytic platforms“.

*My permission to use the name was yanked after this post was largely drafted. I’m sufficiently pleased with the forthcoming offering itself that I can’t get upset about the procedural confusion.

With a couple of exceptions, the statistics/predictive analytics situation seems more reasonable. Industry leaders such as SAS Institute and SPSS (now an IBM company) have engaged in varying degrees of academic outreach. R is in the process of crossing over from academia to business.

Unfortunately, I know next to nothing about Stata or, elsewhere in the technical languages area, Mathworks/Matlab. (Who knew that Mathworks was a $600 million company, local to my geographical area?)

One statistical tool that should perhaps be more present in academia is KXEN. KXEN seems to have some nice differentiation in not making you understand in advance which of your variables are most important. Econometricians and others with large numbers of independent variables might wish to take note.

If you think the true situation is nonlinear, and you’re trying to approximate it with linear models, you almost always have a large number of variables to consider. True, monomials in independent variables aren’t actually independent, but it might be interesting to pretend that they are and see if any insights fall out that could help in more rigorous analysis.

I’d further argue that, as part of neglecting commercial analytic DBMS, the scientific community in particular neglects the potential of integrated analytic platforms. Admittedly, the early leaders in that area — Aster Data, perhaps followed by Netezza (now an IBM company) — aren’t exactly priced in an academic-friendly way. But Vertica, EMC Greenplum, et al. are playing catch-up with analogous technology, and they’re more likely to offer appealing academic pricing.

There’s also the investigative analytics side of business intelligence, especially in the area of visualization/discovery. While Spotfire (now a TIBCO company) got much of its start in research-oriented areas, the otherwise more visible — no pun intended — QlikTech and Tableau don’t seem to have done much in academia. Datameer and yet-younger Hadoop-oriented business intelligence startups don’t seem to be doing much on the academic front either, more’s the pity.

Frankly, I think that most scientific analytic technology needs are also found in the business world.* That convergence will only get closer as businesses focus more on machine-generated data. Commercial software companies should pay more attention to scientists, and scientists should gaze out more often from their ramshackle, budget-constrained ivory towers.

*The converse isn’t as true. Businesses have issues not well reflected in science, derived (for example) from the complexity of their transactional schemas, or from office-politics considerations around “one version of the truth”.

Edit: Some links that seem relevant to this year’s XLDB program

• Zynga and LinkedIn
• Objectivity Infinite Graph
• eBay as of last year’s XLDB (the most expensive blog post I ever wrote, in light of Greenplum’s subsequent response)

• Platform Computing started ~20 years ago.
• Platform Computing claimed close to $100 million in revenue and >500 people.
• (This is Platform Computing’s most famous splash to date.) Platform Computing technology underlies SAS Institute’s preferred method of parallelization, which may variously be called:
  • SAS Grid Manager (the more or less official brand name).
  • SAS HPA (High Performance Analytics), sort of an alternate brand name.
  • MPI (Message Passing Interface), the industry’s name for the underlying semantics/syntax/API.
• Platform Computing’s original business was scientific grid computing.
• Platform Computing’s second major business was its “Symphony” product line. According to Platform Computing, Symphony:
  • Debuted 6-7 years ago.
  • Is more commercially oriented.
  • Is what supports SAS HPA.
  • SAS aside, has been sold to Wall Street and so on.
  • Is sometimes used in conjunction with CEP/streaming, mainly for backtesting.
  • Can be used to build global (parallel) persistent memory for R.
• (This is probably why IBM is buying Platform Computing.) Platform Computing’s has a new MapReduce offering that:
  • Is based on Symphony.
  • Shipped last July, except that early access was a couple months before that.
  • Is focused on:
    • Lowering the latency of MapReduce.
    • Consolidating multiple MapReduce use cases into one high(er)-utilization cluster.
    • Offering workload management in support of those goals.
    • Reliability, availability, predictability, puppies, kittens, and apple pie.
• Is most specifically a MapReduce run-time engine, with other stuff beyond that.

Unfortunately, I’m not precisely clear as to how tied this offering is to Hadoop, but using it with Hadoop is at least the base case. But Platform Computing did say:

• It can support multiple virtual Hadoop clusters, which can be grown or shrunk at will.
• Non-Hadoop workloads can be mixed in.

Platform Computing said that key technical benefits of this offering included:

• 1-3 seconds to start a job, vs. 40-50 in generic Hadoop.
• Automatic recovery of JobTracker nodes.
• Failover for NameNodes.
• Workload management that:
  • Manages all of CPU, I/O, and RAM (this is quickly becoming an industry standard level of capability, although I’m judging more by the standards of the analytic DBMS world).
  • Monitors but doesn’t actively manage network resources.
  • Can reprioritize jobs that are in flight. (Also an industry-standard capability.)

This conflation of scientific, commercial analytic, streaming, and MapReduce is right in IBM’s philosophical wheelhouse. I base that comment on, among other factors:

• How IBM positions “Big Insights”.
• IBM’s “smart consolidation” picture/pitch (which I really should get around to posting).
• The fuss IBM makes about Watson, Blue Gene, and so on.

The IBM acquisition probably obviates a lot of Platform Computing’s previous business comments, but at the time they included:

• POCs (Proofs of Concept):
  • Mainly in financial services, government, and telecom.
  • At both existing customers and new prospects.
  • Typically running 30-50 nodes, 2-50 terabytes.* The smallest databases evidently tended to be an financial services firms.
• Pricing that was starting out:
  • Perpetual license: $3450/server, 21% annual maintenance after the first year.
  • Subscription: $2070/server annually, or $3070 with HDFS support bundled in.

*1 terabyte or less per node is probably the lowest data-per-node figure I’ve heard for anything Hadoop-like — even below Hadapt, and well below what Cloudera and Hortonworks usually see.

The only reason I wouldn’t go is if that turned out to be a lousy week for me to travel to California.

The people who go XLDB tend to be really smart — either research scientists, hardcore database technologists, or others who can hold their own with those folks. Audience participation can be intense; the most talkative members I can recall were Mike Stonebraker, Martin Kersten, Michael McIntire, and myself. Even the vendor folks tend to the smart — past examples include Stephen Brobst, Jeff Hammerbacher, Luke Lonergan, and IBM Fellow Laura Haas. When we had a datageek bash on my last trip to the SF area, several guys said they were planning to attend XLDB as well.

XLDB stands for eXtremely Large DataBases, and those are indeed what gets talked about there. The lead organizer, Jacek Becla, seems to have started XLDB because he has 100 petabytes of astronomical data to plan for. XLDB is where I learned about how CERN manages what must by now be most of the particle physics data in the world. (Even the metadata for the experiment logs is over 10 terabytes.) Facebook, eBay, and Zynga are all on this year’s program.

XLDB’s focus is expanding a bit from data-management-only to analytic techniques as well; I tried to run a panel last year on analytics-DBMS integration before Jeff got hold of the idea and deleted the “DBMS integration” part. But in any case, I’d expect topics discussed at XLDB to be what even I might willingly label “big data”.

The total number of enterprises in the world paying subscription and license fees that they would regard as being for “Hadoop or something Hadoop-related” probably is not much over 100 right now, but I’d expect to see pretty rapid growth. Beyond that, let’s divide customers into three groups:

• Internet businesses.
• Traditional enterprises ‘ internet operations.
• Traditional enterprises’ other operations.

Hadoop vendors, in different mixes, claim to be doing well in all three segments. Even so, almost all use cases involve some kind of machine-generated data, with one exception being a credit card vendor crunching a large database of transaction details. Multiple kinds of machine-generated data come into play — web/network/mobile device logs, financial trade data, scientific/experimental data, and more. In particular, pharmaceutical research got some mentions, which makes sense, in that it’s one area of scientific research that actually enjoys fat for-profit research budgets.

On the partnering side, I heard things about a Hortonworks conference call that do not seem to have been contradicted by my visit to Hortonworks. Namely, Hortonworks promised prospective partners, such as analytic DBMS vendors, hardware vendors, or large system integrators, that it wouldn’t compete with them, in that Hortonworks pledges not to introduce its own products for at least two years. This is presumably targeted most directly at Cloudera, which has lots of partners, but also some proprietary code of its own. MapR, I’d think, would be the #2 target, but that’s just speculation.

The other big part of Hortonworks’ story is the claim that it holds the axe in Apache Hadoop development. Nobody doubts that a large fraction of the work on Hadoop’s core projects was done by Yahoo employees. Many of those indeed moved to Hortonworks; others left Yahoo earlier; Hadoop creator Doug Cutting is actually at Cloudera. So just how dominant Hortonworks really is in core Hadoop development is a bit unclear. Meanwhile, Cloudera people seem to be leading a number of Hadoop companion or sub-projects, including the first two I can think of that relate to Hadoop integration or connectivity, namely Sqoop and Flume. So I’m not persuaded that the “we know this stuff better” part of the Hortonworks partnering story really holds up.

What I am persuaded of is that the Hadoop platform competition is a good thing. Whichever vendors and projects win will be healthier from having had to outcompete worthy alternatives.

Cloudera can identify 22 CDH (Cloudera Distribution [of] Hadoop) clusters holding one petabyte or more of user data each, at 16 different organizations. This does not count Facebook or Yahoo, who are huge Hadoop users but not, I gather, running CDH. Meanwhile, Eric Baldeschwieler of Hortonworks tells me that Yahoo’s latest stated figures are:

• 42,000 Hadoop nodes …
• … holding 180-200 petabytes of data.

That works out near the low end of the range I came up with for Yahoo’s newest gear, namely 36-90 TB/node. Yahoo’s biggest clusters are little over 4,000 nodes (a limitation that’s getting worked on), and Yahoo has over 20 clusters in total.

Based on those numbers, it would seem that 10 or more of Yahoo’s Hadoop clusters are probably in the petabyte range. Facebook no doubt has a few petabyte-scale Hadoop clusters as well. So we’re probably over 3 dozen petabyte+ Hadoop clusters, just counting Yahoo, Facebook, and CDH users. There surely are others too, running Apache Hadoop without Cloudera’s help.

We also have some more information about the scale of Hadoop usage, and the markets it is being used in, because Omer Trajman of Cloudera kindly wrote the following — lightly edited as usual — for quotation:

The number of Petabyte+ Hadoop clusters expanded dramatically over the past year, with our recent count reaching 22 in production (in addition to the well-known clusters at Yahoo! and Facebook). Just as our poll back at Hadoop World 2010 showed the average cluster size at just over 60 nodes, today it tops 200. While mean is not the same as median (most clusters are under 30 nodes), there are some beefy ones pulling up that average. Outside of the well-known large clusters at Yahoo and Facebook, we count today 16 organizations running PB+ clusters running CDH across a diverse number of industries including online advertising, retail, government, financial services, online publishing, web analytics and academic research. We expect to see many more in the coming years, as Hadoop gets easier to use and more accessible to a wide variety of enterprise organizations.

Omer went on to add:

The biggest number of PB clusters are in the advertising space. I often tell people that every ad you see on the internet touched at least one Hadoop cluster (or the Google equivalent).

Bit bucket

• Kinds of data likely to be included: Logs, other technical/external
• Likely use styles: Staging/ETL, investigative
• Canonical example: Log files in a Hadoop cluster
• Stresses: TCO, scale-out, transform/big-query performance, ETL functionality

With the explosion of machine-generated data has come the need for a place to put it all, sometimes called the big bit bucket. This is like the investigative data mart for big databases, but more poly-structured. In some cases it is focused on data staging and transformation; but it can also be used for analysis in place.

The list of candidate technologies to run your bit bucket starts with Hadoop and Splunk.

Archival data store

• Kinds of data likely to be included: Operational, CDR (call detail record), security log
• Likely use styles: Archival, reporting (for compliance), possibly also investigative
• Examples: Any long-term detailed historical store
• Stresses: TCO, compression, scale-out, performance (if multi-use)

Analytic DBMS vendors have been insulting each other with the claim “that’s just an archival data store,” dating back at least to the first time Greenplum was deployed on an underpowered Sun Thumper system. Perhaps only Rainstor truly embraces the archival positioning, and I’ve become pretty dubious about their technical claims and their company alike.

Still, there’s a legitimate need for data stores — especially relational analytic DBMS that:

• Store data cheaply, with high rates of compression.
• Have decent performance if you do want to query the data.
• May have archiving/compliance-specific features as well.

Along with Rainstor, SAND and SenSage have at least partially targeted that use case. In addition, appliance vendors such as Teradata and Netezza try to have an archive-oriented product version in their lineups.

Outsourced data mart

• Kinds of data likely to be included: All
• Likely use styles: Traditional BI, investigative analytics, staging/ETL
• Examples: Advertising tracking, SaaS CRM
• Stresses: Performance, TCO, reliability, concurrency

Much of what happens in analytic database management can also be outsourced. Some applications that run via SaaS (Software as a Service) are analytic. I’ve had three different clients whose main business is picking marketing targets in various vertical segments; others who wanted to add analytics to what were historically OLTP applications; and others yet who just offered online business intelligence. Also, if your fundamental business is gathering data and reselling it to a variety of user organizations, that’s an analytic data management challenge. The possibilities expand from there.

Data outsourcers are in the IT business, and so their IT development is — hopefully! — more serious and less politically encumbered than at many conventional enterprises. Thus, legacy systems and master data management issues are commonly less prevalent, or at least more aggressively disposed of. The same, up to a point, goes for vendor politics.*  Multitenancy is commonly an issue, as is running in the cloud.

*Even so, there’s often That Guy who doesn’t want to migrate away from Oracle, no matter what.

Vertica gets the nod in a number of these cases; it’s cloud-friendly, and often the problem is naturally columnar. Other columnar products can be good choices too, with added brownie points for Infobright if the shop is MySQL-oriented anyway. Running Netezza or other appliances makes sense mainly if you’re pretty sure you want to keep operating your own data centers, but some data outsourcers are just fine with that assumption.

Operational analytic(s) server

• Kinds of data likely to be included: Customer-centric, log, financial trade
• Likely use styles: Advanced operational analytics
• Examples:
  • Lower latency: Web or call-center personalization, anti-fraud
  • Higher latency: Customer profiling, Basel 3 risk analysis
• Stresses: Performance, reliability, analytic functionality, perhaps concurrency

Even with eight different choices, I need a “catch-all” category; this is it.

Suppose you want to do reasonably sophisticated analytics, then use the results in operations. This is the classical challenge in integrating short-request and analytic processing. There are multiple ways to tackle it, embodying different trade-offs in cost, convenience, or analytic accuracy. If the platform on which you want to run your investigative analytics also has the reliability and concurrency appropriate for mission-critical operations, you’re set. Otherwise, you may want to pipe derived data into a more “industrial-strength” DBMS, ideally the one that runs your operational apps anyway

Another option is to integrate a limited amount of analytics immediately into your short-request processing system. For example, as bad as they are at the kinds of queries that require joins, NoSQL systems are often fast at simple aggregations. As MapReduce/NoSQL integrations mature, that option may not require pumping the data anywhere else for deeper analytics; even if it does, at least you’re starting out with the data in a convenient bit bucket.

Streaming/CEP-centric architectures could come into play as well. And it goes on from there. The possibilities in this last category are just too varied to generalize about.

So did I get them all? Or are there yet other analytic data management use cases that I don’t fit into my eight categories?

Let’s try eight categories instead. While no categorization is ever perfect, these each have at least some degree of technical homogeneity. Figuring out which types of analytic database you have or need — and in most cases you’ll need several — is a great early step in your analytic technology planning. 

Enterprise data warehouse (Full or partial)

• Kinds of data likely to be included: All, but especially operational
• Likely use styles: All
• Canonical example: Central EDW for a big enterprise
• Stresses: Concurrency, reliability, workload management

The enterprise data warehouse (EDW) ideal says that you copy all your data into one place, and drive all decision-making from there. Full EDWs are pipedreams. Still, a partial EDW makes sense for most large enterprises, and many indeed already have one. The first product lines to consider for classical EDWs are Teradata, DB2, Exadata, and maybe Microsoft SQL Server, especially if you’re going to stress concurrency and/or operational use cases.

Traditional data mart

• Kinds of data likely to be included: All
• Likely use styles: Business intelligence, budgeting/consolidation, investigative
• Examples: Reporting servers, planning/consolidation servers, anything MOLAP, etc.
• Stresses: Performance, concurrency, TCO

Whether or not you have something like an enterprise data warehouse, it’s common to have lighter-weight data marts as well. A traditional data mart might drive reports and dashboards. Or it might be specialized for budgeting, planning, and/or consolidation.  Some investigative analytics may be in the mix as well.

Any DBMS that can support an EDW can also support a data mart, but it may not be the most cost-effective way to do so. Columnar DBMS might have more attractive performance and TCO (Total Cost of Ownership); the same goes for Netezza. Some of them — e.g. Sybase IQ and Vertica — have excellent track records in concurrent usage as well. Ted Codd pushed what amounts to MOLAP (Multidimensional OnLine Analytic Processing) systems for these use cases. But relational DBMS commonly do a better job, which is one reason most major MOLAP products have wound up at RDBMS companies.

Investigative data mart — agile

• Kinds of data likely to be included: All, especially customer-centric
• Likely use styles: Investigative
• Canonical example: A few analysts getting a few TB to examine
• Stresses: Ease of setup/load, ease of admin, price/performance

Besides the traditional data mart, there are at least two other kinds. Both are focused on investigative analytics, but they’re differentiated by database size.

If you have just a few analysts,* looking at no more than a few terabytes of data (perhaps even just some gigabytes) — and if that data is “single-subject” and fairly homogenous — your watchwords should be “cheap”, “easy”, and “fast”. You don’t need to invest in much hardware, in expensive software, in much administrative effort (the analysts can be their own DBAs),  nor should you endure much set-up time. Just grab a product, grab some data, and start running queries (or extracts into the statistical tool of your choice).

*If you have dozens or even hundreds of analysts hitting the same database, you’re probably back to the more concurrency-oriented scenarios outlined above.

Infobright is often cost-effective among columnar analytic DBMS. Other vendors might cut you a price break as well. If you have multiple terabytes of data, don’t rule out Netezza’s lowest-end products (even if they’d really rather sell you something bigger). Or, if you’re in the sub-terabyte range, maybe you can get by with an in-memory BI tool such as QlikView, and not do anything special on the DBMS side at all.

Investigative data mart — big

• Kinds of data likely to be included: All, especially customer-centric, logs, financial trade, scientific
• Likely use styles: Investigative
• Canonical example: Single-subject 20 TB – 20 PB relational database
• Stresses: Performance, scale-out, analytic functionality

But if you’re looking at tens of terabytes of relational data, or even more, you really do have a “big data” problem. Performance and scalability are major challenges, usually best addressed by MPP (Massively Parallel Processing) systems, such as Netezza, Vertica, Aster Data, ParAccel, Teradata, or Greenplum. Performance POCs (Proofs Of Concept) are a big part of the buying process. Vendor price negotiations are crucial too.

Actually, in the low tens of terabytes you might be able to get away with a shared-disk system that has excellent compression — e.g., columnar products like Sybase IQ, Infobright, or SAND, rather than just Vertica and ParAccel.

Assuming you have affordable, scalable query performance, the competitive differentiator can switch to additional analytic functionality. Aster, Netezza, ParAccel, Vertica, and Greenplum either offer full analytic platforms, or seem to be on the path to doing so. Teradata, which now owns Aster Data, offers substantial built-in analytic capability in its traditional products as well, and the same goes for Sybase IQ.

Continued in Part 2, where we cover some of the more difficult use cases.

• XLDB 4 had a good crowd, including Phil Bernstein (quiet), Mike Stonebraker (not quiet), Martin Kersten (ditto), Luke Lonergan (ditto), Todd Walter (almost unrecognizable without his usual cowboy gear), Oliver Ratzesberger, and a bunch of actual science types.
• XLDB 4 had one weakness — panels with lots of participants, but only a single microphone among them. That tends to make for serial declamations more than true interactive discussion, at least until the audience starts chiming in, which thankfully it tends to eventually do. (I had the same problem in spades while moderating the Boston Big Data Summit panel last year; at least at XLDB 4 nobody was TRYING to filibuster.)

My notes have unfortunately disappeared, but from memory:

• Mike Stonebraker asserted that SciDB outperforms sharded MySQL by two orders of magnitude for some classes of scientific application. One of the big reasons was that SciDB lets you overlap partitions, so that for any feature you want to extract, you can be confident there’s at least one partition that actually contains it.
• I chatted with Peter Breunig of Chevron about analytic issues in the oil & gas industry. I got the impression:
  • Refineries are generally well-instrumented with sensors.
  • Oil wells may not be, especially the less valuable/lower producing ones.
  • He’d love to scatter passive sensors all around, waiting for natural tremors — as opposed to just geologist-set explosions — to provide more insight into what’s under the ground.
  • 50-100 TB geological data sets are common. Processing them takes 2-3 weeks. As the technology gets better, so do the results (rather than the time being shortened).
  • All this suggests that there’s a huge need for better technology in resovoir analysis.
  • His other big unmet analytic desire is refinery simulation.
• Kevin Winsen told about the proposed radio astronomy project ASKAP, which will have raw data volumes that make the LSST’s look small. (More precisely, ASKAP is the name proposed by Australia, one of the two finalists for the location; South Africa presumably has a different name for it.) 8 petabytes/day were mentioned, although most of this will be rapidly discarded. That could be the largest unclassified data acquisition rate out there, although it’s known that there’s a classified one at >10 PB/day (image data).
• Health care researchers repeatedly complained that privacy regulations get in the way of them using clinical data for medical research. Just more grist for my “HIPAA must die so that people can live” mill.
• Mike Stonebraker is pushing the idea of a “science benchmark.” (A paper on same has been posted.) The idea is that the existence of said benchmark should provide a spur for DBMS vendors to make their products run faster for scientific purposes, in line with the supposed salutary effects of TPC-A, TPC-B, and TPC-C. Notwithstanding that attendees included Oracle, Microsoft, EMC/Greenplum, Teradata, and Aster Data — with Greenplum, IBM, and Aster Data also being sponsors — I am skeptical because:
  • Leaders of the XLDB effort seem convinced that only open source DBMS can meet their needs.
  • They further characterize scientific DBMS as a zero billion dollars/year market.

• Google has published about a project called Dremel. Per Tasso Agyros, one of Dremel’s key concepts is nested data structures.
• Those arrays that the XLDB/SciDB folks keep talking about are meant to be nested data structures. Scientific data is of course log-oriented. eBay was very interested in that project too.
• Facebook’s log files have a big nested data structure flavor.

I don’t have a grasp yet on what exactly is happening here, but it’s something.