• Facebook et al. are in effect Software as a Service (SaaS) vendors, not enterprise technology users. In particular:
  • They have the technical chops to rewrite their code as  needed.
  • Unlike packaged software vendors, they’re not answerable to anybody for keeping legacy code alive after a rewrite. That makes migration a lot easier.
  • If they want to write different parts of their system on different technical underpinnings, nobody can stop them. For example …
  • …  Facebook innovated Cassandra, and is now heavily committed to HBase.
• It makes little sense to talk of Facebook’s use of “MySQL.” Better to talk of Facebook’s use of “MySQL +  memcached  + non-transparent sharding.” That said:
  • It’s hard to see why somebody today would use MySQL +  memcached  + non-transparent sharding for a new project. At least one of Couchbase or transparently-sharded MySQL is very likely a superior alternative. Other alternatives might be better yet.
  • As noted above in the example of Facebook, the many major web businesses that are using MySQL +  memcached  + non-transparent sharding for existing projects can be presumed able to migrate away from that stack as the need arises.

Continuing with that discussion of DBMS alternatives:

• If you just want to write to the memcached API anyway, why not go with Couchbase?
• If you want to go relational, why not go with MySQL? There are many alternatives for scaling or accelerating MySQL — dbShards, Schooner, Akiban, Tokutek, ScaleBase, ScaleDB, Clustrix, and Xeround come to mind quickly, so there’s a great chance that one or more will fit your use case. (And if you don’t get the choice of MySQL flavor right the first time, porting to another one shouldn’t be all THAT awful.)
• If you really, really want to go in-memory, and don’t mind writing Java stored procedures, and don’t need to do the kinds of joins it isn’t good at, but do need to do the kinds of joins it is, VoltDB could indeed be a good alternative.

And while we’re at it — going schema-free often makes a whole lot of sense. I need to write much more about the point, but for now let’s just say that I look favorably on the Big Four schema-free/NoSQL options of MongoDB, Couchbase, HBase, and Cassandra.

Bit bucket

• Kinds of data likely to be included: Logs, other technical/external
• Likely use styles: Staging/ETL, investigative
• Canonical example: Log files in a Hadoop cluster
• Stresses: TCO, scale-out, transform/big-query performance, ETL functionality

With the explosion of machine-generated data has come the need for a place to put it all, sometimes called the big bit bucket. This is like the investigative data mart for big databases, but more poly-structured. In some cases it is focused on data staging and transformation; but it can also be used for analysis in place.

The list of candidate technologies to run your bit bucket starts with Hadoop and Splunk.

Archival data store

• Kinds of data likely to be included: Operational, CDR (call detail record), security log
• Likely use styles: Archival, reporting (for compliance), possibly also investigative
• Examples: Any long-term detailed historical store
• Stresses: TCO, compression, scale-out, performance (if multi-use)

Analytic DBMS vendors have been insulting each other with the claim “that’s just an archival data store,” dating back at least to the first time Greenplum was deployed on an underpowered Sun Thumper system. Perhaps only Rainstor truly embraces the archival positioning, and I’ve become pretty dubious about their technical claims and their company alike.

Still, there’s a legitimate need for data stores — especially relational analytic DBMS that:

• Store data cheaply, with high rates of compression.
• Have decent performance if you do want to query the data.
• May have archiving/compliance-specific features as well.

Along with Rainstor, SAND and SenSage have at least partially targeted that use case. In addition, appliance vendors such as Teradata and Netezza try to have an archive-oriented product version in their lineups.

Outsourced data mart

• Kinds of data likely to be included: All
• Likely use styles: Traditional BI, investigative analytics, staging/ETL
• Examples: Advertising tracking, SaaS CRM
• Stresses: Performance, TCO, reliability, concurrency

Much of what happens in analytic database management can also be outsourced. Some applications that run via SaaS (Software as a Service) are analytic. I’ve had three different clients whose main business is picking marketing targets in various vertical segments; others who wanted to add analytics to what were historically OLTP applications; and others yet who just offered online business intelligence. Also, if your fundamental business is gathering data and reselling it to a variety of user organizations, that’s an analytic data management challenge. The possibilities expand from there.

Data outsourcers are in the IT business, and so their IT development is — hopefully! — more serious and less politically encumbered than at many conventional enterprises. Thus, legacy systems and master data management issues are commonly less prevalent, or at least more aggressively disposed of. The same, up to a point, goes for vendor politics.*  Multitenancy is commonly an issue, as is running in the cloud.

*Even so, there’s often That Guy who doesn’t want to migrate away from Oracle, no matter what.

Vertica gets the nod in a number of these cases; it’s cloud-friendly, and often the problem is naturally columnar. Other columnar products can be good choices too, with added brownie points for Infobright if the shop is MySQL-oriented anyway. Running Netezza or other appliances makes sense mainly if you’re pretty sure you want to keep operating your own data centers, but some data outsourcers are just fine with that assumption.

Operational analytic(s) server

• Kinds of data likely to be included: Customer-centric, log, financial trade
• Likely use styles: Advanced operational analytics
• Examples:
  • Lower latency: Web or call-center personalization, anti-fraud
  • Higher latency: Customer profiling, Basel 3 risk analysis
• Stresses: Performance, reliability, analytic functionality, perhaps concurrency

Even with eight different choices, I need a “catch-all” category; this is it.

Suppose you want to do reasonably sophisticated analytics, then use the results in operations. This is the classical challenge in integrating short-request and analytic processing. There are multiple ways to tackle it, embodying different trade-offs in cost, convenience, or analytic accuracy. If the platform on which you want to run your investigative analytics also has the reliability and concurrency appropriate for mission-critical operations, you’re set. Otherwise, you may want to pipe derived data into a more “industrial-strength” DBMS, ideally the one that runs your operational apps anyway

Another option is to integrate a limited amount of analytics immediately into your short-request processing system. For example, as bad as they are at the kinds of queries that require joins, NoSQL systems are often fast at simple aggregations. As MapReduce/NoSQL integrations mature, that option may not require pumping the data anywhere else for deeper analytics; even if it does, at least you’re starting out with the data in a convenient bit bucket.

Streaming/CEP-centric architectures could come into play as well. And it goes on from there. The possibilities in this last category are just too varied to generalize about.

So did I get them all? Or are there yet other analytic data management use cases that I don’t fit into my eight categories?

Let’s try eight categories instead. While no categorization is ever perfect, these each have at least some degree of technical homogeneity. Figuring out which types of analytic database you have or need — and in most cases you’ll need several — is a great early step in your analytic technology planning. 

Enterprise data warehouse (Full or partial)

• Kinds of data likely to be included: All, but especially operational
• Likely use styles: All
• Canonical example: Central EDW for a big enterprise
• Stresses: Concurrency, reliability, workload management

The enterprise data warehouse (EDW) ideal says that you copy all your data into one place, and drive all decision-making from there. Full EDWs are pipedreams. Still, a partial EDW makes sense for most large enterprises, and many indeed already have one. The first product lines to consider for classical EDWs are Teradata, DB2, Exadata, and maybe Microsoft SQL Server, especially if you’re going to stress concurrency and/or operational use cases.

Traditional data mart

• Kinds of data likely to be included: All
• Likely use styles: Business intelligence, budgeting/consolidation, investigative
• Examples: Reporting servers, planning/consolidation servers, anything MOLAP, etc.
• Stresses: Performance, concurrency, TCO

Whether or not you have something like an enterprise data warehouse, it’s common to have lighter-weight data marts as well. A traditional data mart might drive reports and dashboards. Or it might be specialized for budgeting, planning, and/or consolidation.  Some investigative analytics may be in the mix as well.

Any DBMS that can support an EDW can also support a data mart, but it may not be the most cost-effective way to do so. Columnar DBMS might have more attractive performance and TCO (Total Cost of Ownership); the same goes for Netezza. Some of them — e.g. Sybase IQ and Vertica — have excellent track records in concurrent usage as well. Ted Codd pushed what amounts to MOLAP (Multidimensional OnLine Analytic Processing) systems for these use cases. But relational DBMS commonly do a better job, which is one reason most major MOLAP products have wound up at RDBMS companies.

Investigative data mart — agile

• Kinds of data likely to be included: All, especially customer-centric
• Likely use styles: Investigative
• Canonical example: A few analysts getting a few TB to examine
• Stresses: Ease of setup/load, ease of admin, price/performance

Besides the traditional data mart, there are at least two other kinds. Both are focused on investigative analytics, but they’re differentiated by database size.

If you have just a few analysts,* looking at no more than a few terabytes of data (perhaps even just some gigabytes) — and if that data is “single-subject” and fairly homogenous — your watchwords should be “cheap”, “easy”, and “fast”. You don’t need to invest in much hardware, in expensive software, in much administrative effort (the analysts can be their own DBAs),  nor should you endure much set-up time. Just grab a product, grab some data, and start running queries (or extracts into the statistical tool of your choice).

*If you have dozens or even hundreds of analysts hitting the same database, you’re probably back to the more concurrency-oriented scenarios outlined above.

Infobright is often cost-effective among columnar analytic DBMS. Other vendors might cut you a price break as well. If you have multiple terabytes of data, don’t rule out Netezza’s lowest-end products (even if they’d really rather sell you something bigger). Or, if you’re in the sub-terabyte range, maybe you can get by with an in-memory BI tool such as QlikView, and not do anything special on the DBMS side at all.

Investigative data mart — big

• Kinds of data likely to be included: All, especially customer-centric, logs, financial trade, scientific
• Likely use styles: Investigative
• Canonical example: Single-subject 20 TB – 20 PB relational database
• Stresses: Performance, scale-out, analytic functionality

But if you’re looking at tens of terabytes of relational data, or even more, you really do have a “big data” problem. Performance and scalability are major challenges, usually best addressed by MPP (Massively Parallel Processing) systems, such as Netezza, Vertica, Aster Data, ParAccel, Teradata, or Greenplum. Performance POCs (Proofs Of Concept) are a big part of the buying process. Vendor price negotiations are crucial too.

Actually, in the low tens of terabytes you might be able to get away with a shared-disk system that has excellent compression — e.g., columnar products like Sybase IQ, Infobright, or SAND, rather than just Vertica and ParAccel.

Assuming you have affordable, scalable query performance, the competitive differentiator can switch to additional analytic functionality. Aster, Netezza, ParAccel, Vertica, and Greenplum either offer full analytic platforms, or seem to be on the path to doing so. Teradata, which now owns Aster Data, offers substantial built-in analytic capability in its traditional products as well, and the same goes for Sybase IQ.

Continued in Part 2, where we cover some of the more difficult use cases.

*Actually, he didn’t admit that what he was advocating was a different kind of DBMS, namely a MOLAP one — but he was. And he was wrong anyway about the necessity for MOLAP. But let’s overlook those details.

Nonetheless, relational DBMS dominate the market. As I see it, the reasons for relational dominance cluster into four areas (which of course overlap):

• Data re-use. Ted Codd’s famed original paper referred to shared data banks for a reason.
• The benefits of normalization, which include:
  • You only have to do programming work of writing something once …
  • … and you don’t have to do the programming work of keeping multiple versions of the information consistent.
  • You only have to do processing work of writing something once.
  • You only have to buy storage to hold each fact once.
• Separation of concerns.
  • Different people can worry about programming and “database stuff.”
  • Indeed, even performance optimization can sometimes be separated from programming (i.e., when all you have to do to get speed is implement the correct indexes).
• Maturity and momentum, as reflected in the availability of:
  • People.
  • A broad variety of mature relational DBMS.
  • Vast amounts of packaged software that “talks” SQL.

Generally speaking, I find the reasons for sticking with relational technology compelling in cases such as: 

• You’re building a low-volume, medium-complexity suite of applications that will evolve over time. This is the use case for which relational DBMS were invented, and they’re still great for it.
• Your (duplicated) data volumes would be ridiculous if you didn’t do a reasonable amount of normalization. Once you need to normalize, you need to do joins — and if you’re doing joins, you’re in relational territory.
• You simply don’t see a cost/benefit advantage to moving away from proven legacy technology. If you’re looking for an off-the-shelf answer to your needs — or if you’re inventorying your own technological shelves — relational-oriented technology has overwhelming share.

For many enterprises, that third point alone should be decisive in a large fraction of cases.

But the advantages of relational technology are less clear when you’re doing serious engineering of path-breaking new applications, where by “serious engineering” I mean:

• The problem is big enough that you simply want the best solution, with only loose coupling needed to the rest of your technical environment.
• Long-lasting “strategic” or legacy technology is not a great concern; you’re willing to keep “rebuilding the 747 while it’s flying” if that’s what’s necessary to get the best possible result.
• You have access to sufficient quantities of sufficiently smart people.

For example:

• I recently suggested that innovative SaaS vendors could adopt object-oriented database technology.
• Major web applications are rarely very relational. Until recently, the default approach to scaling out web databases was memcached/sharded MySQL, hardly a whole-hearted adoption of relational technology. Now NoSQL DBMS are vigorous competitors.
• Analytic challenges that amount to teasing out signals from streams of data are sometimes handled non-relationally as well, although it’s often nice to be able to do a few joins to mix in information from more relationally-structured data.

Not coincidentally, in a lot of those cases, throwing performance concerns “over the wall” to the database administrator isn’t going to work.

*I do expect the pendulum to swing back a bit as high-performance/highly-scalable MySQL implementations mature, but there are relatively few supporting examples to date.

To look at it another way, it’s right to be skeptical about relational DBMS when you can defeat all of the reasons to favor them. For example:

• Data re-use may not arise when applications are self-contained and rapidly-changing.
• Sometimes you don’t need to normalize your data.
• It’s not obvious that the relational approach to separation of concerns is the best one. Perhaps you’d be better off with the people who understand a specific application best being responsible for all the decisions connected with it.
• As for that maturity and momentum:
  • People don’t actually learn much SQL in school.
  • Are any of the mature relational DBMS what you really want?
  • Is any of that packaged software out there really helpful for your specific problem?

I should probably stop there. But in an appeal to authority, I’ll close instead with a quote from Codd’s own OLAP paper:

IT should never forget that technology is a means to an end, and not an end in itself. Technologies must be evaluated individually in terms of their ability to satisfy the needs of their respective users. IT should never be reluctant to use the most appropriate interface to satisfy users’ requirements. Attempting to force one technology or tool to satisfy a particular need for which another tool is more effective and efficient is like attempting to drive a screw into a wall with a hammer when a screwdriver is at hand: the screw may eventually enter the wall but at what cost?

Related link

• My exchange with Mike Stonebraker highlighting our shared advocacy for database diversity

Below please find three excerpts* that convey the essence of my thinking on NoSQL. For much more detail, please see the article itself.

*Notwithstanding my admiration for Doug’s editing, the excerpts are taken from my final pre-editing submission, not from the published article itself.

My quasi-definition of “NoSQL” wound up being: 

NoSQL DBMS start from three design premises:

• Transaction semantics are unimportant, and locking is downright annoying.
• Joins are also unimportant, especially joins of any complexity.
• There are some benefits to having a DBMS even so.

NoSQL DBMS further incorporate one or more of three assumptions:

• The database will be big enough that it should be scaled across multiple servers.
• The application should run well if the database is replicated across multiple geographically distributed data centers, even if the connection between them is temporarily lost.
• The database should run well if the database is replicated across a host server and a bunch of occasionally-connected mobile devices.

In addition, NoSQL advocates commonly favor the idea that a database should have no fixed schema, other than whatever emerges as a byproduct of the application-writing process.

I subdivided the space by saying:

If not SQL, then what?  A number of possibilities have been tried, with the four main groups being:

• Simple key-value store.
• Quasi-tabular.
• Fully SQL/tabular.
• Document/object.

DBMS based on graphical data models are also sometimes suggested to be part of NoSQL, as are the file systems that underlie many MapReduce implementations. But as a general rule, those data models are most effective for analytic use cases somewhat apart from the NoSQL mainstream.

My conclusion was:

So should you adopt NoSQL technology? Key considerations include:

• Immaturity. The very term “NoSQL” has only been around since 2009. Most NoSQL “products” are open source projects backed by a company of fewer than 20 employees.
• Open source. Many NoSQL adopters are constrained, by money or ideology, to avoid closed-source products. Conversely, it is difficult to deal with NoSQL products’ immaturity unless you’re comfortable with the rough-and-tumble of open source software development.
• Internet orientation. A large fraction of initial NoSQL implementations are for web or other internet (e.g., mobile application) projects.
• Schema mutability. If you like the idea of being able to have different schemas for different parts of the same “table,” NoSQL may be for you. If you like the database reusability guarantees of the relational model, NoSQL may be a poor fit.
• Project size. For a large (and suitable) project, the advantages of NoSQL technology may be large enough to outweigh its disadvantages. For a small, ultimately disposable project, the disadvantages of NoSQL may be minor. In between those extremes, you may be better off with SQL.
• SQL DBMS diversity. The choice of SQL DBMS goes far beyond the “Big 3-4” of Oracle, IBM DB2, Microsoft SQL Server, and SAP/Sybase Adaptive Server Anywhere. MySQL, PostgreSQL, and other mid-range SQL DBMS – open source or otherwise – might meet your needs. So might some of the scale-out-oriented startups cited above. Or if your needs are more analytic, there’s a whole range of powerful and cost-effective specialized products, from vendors such as Netezza, Vertica, Aster Data, or EMC/Greenplum.

Bottom line: For cutting-edge applications – often but not only internet-centric — NoSQL technology can make sense today. In other use cases, its drawbacks are likely to outweigh its advantages.

Related link

• How to tell when you need ACID-compliant transaction integrity
  

• There are a lot of big datasets out there, where “big” commonly means “petabyte scale.”
• Owners of that much data commonly like to store it using free or quasi-free software, especially if the data isn’t structured in such a way that relational tables are a great fit in the first place. HDFS (Hadoop Distributed File System) is the default choice. (Of course, there always are exceptions.)
• Some kinds of analytics can be done perfectly well in Hadoop.
• Some kinds of analytics, of course, can not be done well in Hadoop, with the most obvious examples being:
  • Queries that involve serious joins.
  • Anything that requires a lower latency than Hadoop provides.
• When doing analytics in Hadoop on data stored in HDFS, you often will want to include data you’re storing in your relational DBMS.

So Cloudera is promising fast, bidirectional connectors between Hadoop/HDFS and various DBMS, such as Aster Data nCluster, and will provide them on a services basis even before the productized versions ship. Here “fast” should and in multiple cases does mean “fully parallel,” with all data-owning nodes on either side (Hadoop or HDFS) more or less equally involved. Indeed, Aster is (I think for the first time) bypassing its loader nodes, instead sending Hadoop data straight to its workers.

• Manage data to high standards of accuracy, consistency, cleanliness, clarity, and security.
• Manage all the data in your organization.

Pick ONE.

There’s little to dislike in the enterprise data warehouse dream, as represented (for example) in this 2004 Teradata Magazine article. But in a world where ever more data comes in from ever more sources – and is needed ever faster – it simply isn’t realistic to expect that all an enterprise’s data will be vetted, organized, and managed to the highest of standards.

This is a core premise of Greenplum’s Enterprise Data Cloud (EDC)/Chorus marketing initiative, and in that respect Greenplum is correct.

If the EDW is a great idea that can never be 100% implemented, what should you do? At conventional enterprises, the answer is pretty obvious: Manage some of your data to enterprise data warehouse standards, but not all of it. Specifically, your highest-value data should be in something that looks like a classic enterprise data warehouse, and your lower-value data shouldn’t.

Of course, if you’re a data mart outsourcer or other analytic service provider, whose data is about your customers’ businesses rather than your own, and whose business is managing your customers’ data, this may not apply to you. But otherwise it’s a position with many supporting arguments, including:

• Financial reporting, compliance, and other legitimate concerns introduce rigidity into data models. This increases the cost and reduces the speed of getting data into enterprise data warehouses.
• Data governance procedures imposed for any other business purpose have the same effect. What’s deemed necessary for enterprise data warehouses can be fatal to timely analytics.
• The highest-value data typically comes from transactional systems, such as order entry or sales contact management. So it starts out with a degree of governance that, say, web log files may never enjoy.
• In some enterprises, it is affordable or even cost-effective to manage your highest-value data in your favorite big-brand DBMS, but necessary to manage most of your data in something with lower TCO (Total Cost of Ownership). Big-brand OLTP DBMS are often better (or at least less bad) at managing enterprise data warehouses than they are at running data mart workloads.
• At certain enterprise and database sizes, it may indeed make sense to run what amounts to an enterprise data warehouse out of the same database instance that does OLTP, while putting larger data sets into more cost-effective data marts. A trend to “operational BI” may actually make that option more appealing going forward than it has been in the past.
• And finally, there’s the empirical fact that not one really large enterprise on the whole planet has a true, perfectly comprehensive enterprise data warehouse. At least, I’ve never heard of one.

Related links

• Even Teradata doesn’t push an EDW-only strategy any more
• I agreed when Greenplum first started pushing the EDC idea that something like it would be the future of data marts

• No technology category name is ever perfect.
• It’s particularly hard to describe NoSQL (Not Only SQL) accurately, given the basic confusion as to what NoSQL is all about.
• That said, it seems pretty clear that NoSQL is about making big websites (and perhaps other cloud-like installations) run and scale.
• Dwight Merriman (founder/CEO of MongoDB vendor 10gen) is heading in the right direction when he says that the unifying ideas of NoSQL are that you do away with transactions and joins. But if he’s ever said something like “NoSQL is Foo without joins and transactions,” I don’t know what Foo is.
• Actually, I do know what Foo is – Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. I just don’t know what Foo is called.
• Obviously, Foo is a lot like OLTP (OnLine Transaction Processing). However, it would be pretty silly for Foo to actually be OLTP, given that one of the core points of NoSQL is that you don’t have transactions.
• It not just the “T” part of OLTP that’s fried. Calling something “OnLine” only makes sense as long as offline is an option, and offline transaction processing has been obsolete for a very long time.*

*Sure, if you strain you can talk yourself into exceptions. But the point stands.

So we need a name for Foo, where Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. Thus, three major subcategories of more-or-less disk-based Foo are:

• No-compromises ACID-compliant relational OLTP
• Sharded MySQL
• NoSQL

There may be some more purely memory-centric versions too, but let’s put those aside for the moment.

Absent a better idea, I can squeeze Foo into yet another four-letter acronym:

HVSP (High-Volume Simple Processing)

That’s as imperfect as any other category name, and an awkward mouthful to boot. So I’d love to hear a better one; if you have such, please share it! In the mean time, I think “HVSP” has merit because:

• The “Processing” part should be noncontroversial.
• “High-Volume” is inherent to the challenge. If RDBMS scale well enough for your use case, using something less powerful is probably silly.* Similarly, while Oracle shines at high-volume OLTP workloads, there are many cheaper DBMS that do a fine job of OLTP at lower volumes.
• “Simple” is the core principle of NoSQL systems, which drop joins and transactions as being too much foofarah. That only makes sense at all under the assumption that you have bone-simple queries and updates, so that programming around the lack of joins and transactions isn’t all that much of a burden.
• Something similar is true of sharded MySQL.
• Less obviously, “simple” is a core principle of relational OLTP as well. The point of the relational model is to cap the complexity of data operations, or more precisely to hide that complexity from programmers.
• And overloading the word “simple” a bit, it’s fair to say that if you’re reading or writing one record at a time, you’re doing something relatively simple, at least as opposed to what you do in analytic processing. The OLTP vs. OLAP distinction is preserved in this name change.
• The whole thing matches my definition above, namely “what happens when lots of people want to get small amounts each of information in or out of a database at the same time.”

*Assuming, of course, that rows-and-tables are a good metaphor for your data structure in the first place.

Systems I’m leaving out of the HVSP and hence also NoSQL categories include:

• Hadoop and other batch-oriented MapReduce. Hadoop isn’t part of NoSQL. I’m pretty sure that Cloudera CEO Mike Olson agrees with me.
• More generally, non-SQL data stores that don’t meet the HVSP criteria. Dave Kellogg stretches things when he claims that MarkLogic is a NoSQL system. (But then, that was in a post where he seemingly praised a train wreck of an article.)

But hey – what good is a categorization if it doesn’t leave some things out?

• Traditional data of the sort that’s stored in relational databases, aka “structured.”
• Everything else, aka “unstructured” or “semi-structured” or “complex.”

There are plenty of problems with these formulations, not the least of which is that the supposedly “unstructured” data is the kind that actually tends to have interesting internal structures. But of the many reasons why these distinctions don’t tend to work very well, I think the most important one is that:

Databases shouldn’t be divided into just two categories. Even as a rough-cut approximation, they should be divided into three, namely:

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays
• Human/Nontabular data — i.e., all other data generated by humans
• Machine-Generated data

Even that trichotomy is grossly oversimplified, for reasons such as:

• These categories overlap.
• There are kinds of data that get into fuzzy border zones.
• Not all data in each category has all the same properties.

But at least as a starting point, I think this basic categorization has some value.

By human-generated data that fits well into relational tables or arrays, what I really mean is: the input from most conventional kinds transactions – purchase/sale, inventory/manufacturing, employment status change, etc. This is the core data managed by OLTP relational DBMS everywhere. It is also the core data in analytic relational or MOLAP databases. The vast majority of what we think or know about “database management” applies primarily to data of this kind, in large part because of two fundamental properties of this information:

• It is meaningful to contemplate this data as being 100% accurate and complete (even if that goal is difficult to achieve in the real world).
• This data is precise – i.e., one can check predicates against it and (give or take regrettable data imperfections) get inarguable yes/no answers.

For most enterprises, this is the most important data they have. It was created as a result of expensive business activities. It deals directly with money, employees, physical goods, and the rest of the things that make an enterprise go. It can be fruitfully analyzed in ever more ways, which is why it should never be thrown out or even entirely relegated to tape, now that data warehouse software, hardware, and storage has become so cheap. (“Disk is the new tape.”) And because of the importance of both preserving and accessing it, it should often be stored in multiple copies – OLTP, data warehouse, data mart, in-memory analytics, near-line quasi-archive, MOLAP cubes (if you must) and so on, plus of course replicas for high throughput and availability.

But humans generate many other kinds of data as well, especially in a form directly suitable for communication – text (in many formats), documents (text or otherwise), pictures, videos, etc. Traditional relational databases are a poor home for this kind of data because:

• This data often deals with opinions or aesthetic judgments – there is little concept of perfect accuracy.
• Similarly, there is little concept of perfect completeness.
• There’s also little concept of perfectly, unarguably accurate query results – different people will have different opinions as to what comprises good results for a search.
• Queries don’t lend themselves to binary answers; rather, documents can have differing degrees of relevancy.

Systems for managing this kind of data are much less advanced than relational database managers. Nobody knows how to get all the information out of a text document, or query all of it if they could, and the story is even worse for non-text examples. The systems that give the best query results aren’t necessarily the same ones that have the best database administration features. Basically, this area is still a mess, and it’s a mess that consumes a huge fraction of all the data storage products sold today.

But give or take questions of storage efficiency and deduplication, if humans created that kind of data, they put a lot of effort into it, so it’s worth keeping. Besides, compliance regulations commonly mandate that we do so – except, perhaps, when they mandate that we throw it away.

Machine-generated data is a whole other can of worms. Paradigmatic examples of what I mean by machine-generated data include:

• Computer, network, and other equipment logs
• Satellite and similar telemetry (whether for espionage or science)
• Location data such as RFID chip readings, GPS system output, etc.
• Temperature and other environmental sensor readings
• Sensor readings from factories, pipelines, etc.
• Output from many kinds of medical device, in hospitals and (increasingly) homes alike

Unlike human-generated data, whose growth is constrained by macro factors such as population and total level of economic activity, machine-generated data will continue to grow as fast as Moore’s Law lets it. That fact has two profound consequences:

• It is unrealistic to hope ever to keep most or all machine-generated data, whereas I think that’s exactly what should and will happen with human-generated data
• Before long, most data (by volume) will be machine-generated

And so it is not really an exaggeration to say that machine-generated data is the future of data management.

I’d like to close this long post by immediately pointing out some of the flaws in this simple trichotomy. One obvious gray area lies in hybrid human/machine-generated data, three big examples of which are:

• Web clickstreams
• Call detail records (CDR)
• Stock trades

In all three cases, we are quickly getting to the point where this data is preserved in its entirety (even if the network event data associated with the web logs is reduced before storage). And in each case it fits pretty well into RDBMS, although Hadoop has a role to play as well. So pretending it’s purely human-generated probably isn’t all that misleading.

Another gray area lies in text that gets linguistically processed – i.e. via text-mining tools – with the output placed into a relational database. I don’t immediately see a workaround for that flaw in my labeling scheme.  So let’s just say no taxonomy is perfect.*

*Come to think of it, that’s one of the problems holding back text-mining technology.

And of course some of the NoSQL folks would note that I was oversimplifying when I tied my first category specifically to relational DBMS. So would the folks at Intersystems.

But the biggest oversimplification stems from this:

As Mike Stonebraker* and I argued a couple of years ago, I really think that database management technologies should be divided into 10+ categories.

*Note: The links to Stonebraker’s own posts will be broken until Vertica’s webmaster gets his/her act together. But you can find them under other URLs via web search.

]]> http://www.dbms2.com/2010/01/17/three-broad-categories-of-data/feed/ 19 http://www.dbms2.com/2009/12/12/legit-nosql-key-value-store/ http://www.dbms2.com/2009/12/12/legit-nosql-key-value-store/#comments Sat, 12 Dec 2009 06:06:52 +0000 Curt Monash http://www.dbms2.com/?p=1319 I’ve written some snarky things about the “NoSQL” concept – or at least the moniker. (Carl Olofson’s term “non-schematic databases” seems less bad.) Yet I’m actually favorable about the increasing use of SQL alternatives. Perhaps I should pull those thoughts together.

Relational database management systems were invented to let you use one set of data in multiple ways, including ways that are unforeseen at the time the database is built and the first applications against it are written. In almost all cases, RDBMS are the best way to manage data of that nature. The increasing diversity in kinds of RDBMS – especially on the analytic side – just strengthens the point: Also, RDBMS are more mature than most competing technologies. And so, for multiple reasons, your highest-value data often belongs in an RDBMS.

The main reason I wrote “often” instead of “always” is that some of your highest-value data is in formats that don’t fit well into an RDBMS at all. The most obvious example is text. Text data shouldn’t be shoehorned into the relational model, and to date it often has been best to manage text entirely outside of RDBMS.

Even lower-value data often belongs in RDBMS. eBay has huge volumes of log files stored in RDBMS. Yahoo and Facebook both prefer Hadoop over traditional RDBMS – but both are also building capabilities into Hadoop that pretty much will amount to a new RDBMS.

Science provides some pretty compelling use cases for non-SQL-oriented DBMS. So does health care. But that’s not the kind of thing the NoSQL folks seem to focus on. Rather, “NoSQL” seems mainly to encompass three kinds of systems:

• Key-value stores, such as Cassandra or BigTable. So far as I can tell, a key-value store is just a substitute for a transaction-processing DBMS, inferior in most ways except scalable performance, where they can shine. As an additional benefit, a key-value store frees you from that pesky SQL programming you never learned in school. What’s more, if you can’t stabilize your schema, a key-value store lets you get some level of database programming done anyway.
• Document managers such as CouchDB or MongoDB. I haven’t figured out how those are different from low-volume distributed file systems, or why anybody should care about them.
• DBMS imitations built on top of HDFS (Hadoop Distributed File System). For the most part, I think those will wind up talking SQL.

So it seems that, at least for now, the legit part of the NoSQL movement is the distributed key-value stores. Frankly, even if transactional data is persisted in a key-value store, it should wind up in an RDBMS, whether OLTP or analytic. But even so, the big web companies seem to have demonstrated that key-value stores have very legitimate uses.