NoSQL = HVSP (High Volume Simple Processing) without joins or explicit transactions

Within that realm, Dwight offered a two-part taxonomy of NoSQL systems, according to their data model and replication/sharding strategy. I’d be happier, however, with at least three parts to the taxonomy:

• How data looks logically on a single node
• How data is stored physically on a single node
• How data is distributed, replicated, and reconciled across multiple nodes, and whether applications have to be aware of how the data is partitioned among nodes/shards.

After talking with Dwight, and also with Cassandra project chair Jonathan Ellis, I feel I’m doing decently in understanding the first of those three areas. But there’s a long way yet to go on the other two.

In Dwight’s opinion, as I understand it, NoSQL data models come in four general kinds.

• Key-value stores, more or less pure. I.e., they store keys+BLOBs (Binary Large OBjects), except that the “Large” part of “BLOB” may not come into play.
• Table-oriented, more or less. The major examples here are Google’s BigTable, and Cassandra.
• Document-oriented, where a “document” is more like XML than free text. MongoDB and CouchDB are the big examples here.
• Graph-oriented. To date, this is the smallest area of the four. I’m reserving judgment as to whether I agree it’s properly included in HVSP and NoSQL.

As Dwight sees it, JSON (JavaScript Object Notation) is the emerging markup standard for the document-oriented data models, and to some extent the BLOB part of key-value models as well. Reasons seem to include:

• JSON is something web developers are likely to know anyway.
• JSON, unlike XML, is schema-less. In the NoSQL world, that’s perceived as a good thing.
• Perhaps for both these reasons, JSON is perceived as easier to use than XML.

Except as noted, I’m not aware of anything that solidly contradicts the above.

Dwight went on to say that there are two main NoSQL replication/sharding models, in line with the seminal papers to which I previously linked:

• Based on or resembling Dynamo. The core idea here is accepting eventual consistency among nodes as being good enough, even if that means you sometimes read dirty data. The benefit is that you never are blocked from writing. By way of contrast, systems that enforce true inter-node consistency (think of a two-phase commit) can shut you down from writing if consistency guarantees aren’t being confirmed in a timely manner. Thus, in a Dynamo-like scheme you write data to multiple nodes, via consistent hashing; then when the time comes you read one or more nodes, and hope that what you’re getting back is a correct result.
• Based on or resembling BigTable. In this model you’re trying to keep the nodes fully consistent in the usual way, e.g. by synchronous replication. Indeed, what’s being kept consistent is both data itself, and metadata about the data’s location. Details surely vary a lot from implementation to implementation.

I’m fuzzier on this stuff than on the data models, because to date nobody has ever explained to me how an actual live system (MongoDB, Cassandra, whatever) implements its replication strategy. Also, while I think that in both these models applications are allowed to be ignorant of the replication/sharding strategy, I’m not as sure of that as I’d like to be.

If we stop here, we already have something useful. MongoDB has a document data model, and is in the BigTable-like replication camp, at least at first. Cassandra has a table-like data model, and is on the Dynamo-like eventual consistency side. But to say those are the only differences that matter would be like saying that all shared-disk RDBMS (e.g., Oracle and Sybase IQ) are essentially alike. That, of course, would be nonsense.

So a third dimension needed in this taxonomy is how the systems actually bang data on and off of disk (or silicon, as the case may be). I don’t yet have an overview of that. I know something of how Cassandra does it, and will write about same in a future post, but that’s about it. So please stay tuned.

• No technology category name is ever perfect.
• It’s particularly hard to describe NoSQL (Not Only SQL) accurately, given the basic confusion as to what NoSQL is all about.
• That said, it seems pretty clear that NoSQL is about making big websites (and perhaps other cloud-like installations) run and scale.
• Dwight Merriman (founder/CEO of MongoDB vendor 10gen) is heading in the right direction when he says that the unifying ideas of NoSQL are that you do away with transactions and joins. But if he’s ever said something like “NoSQL is Foo without joins and transactions,” I don’t know what Foo is.
• Actually, I do know what Foo is – Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. I just don’t know what Foo is called.
• Obviously, Foo is a lot like OLTP (OnLine Transaction Processing). However, it would be pretty silly for Foo to actually be OLTP, given that one of the core points of NoSQL is that you don’t have transactions.
• It not just the “T” part of OLTP that’s fried. Calling something “OnLine” only makes sense as long as offline is an option, and offline transaction processing has been obsolete for a very long time.*

*Sure, if you strain you can talk yourself into exceptions. But the point stands.

So we need a name for Foo, where Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. Thus, three major subcategories of more-or-less disk-based Foo are:

• No-compromises ACID-compliant relational OLTP
• Sharded MySQL
• NoSQL

There may be some more purely memory-centric versions too, but let’s put those aside for the moment.

Absent a better idea, I can squeeze Foo into yet another four-letter acronym:

HVSP (High-Volume Simple Processing)

That’s as imperfect as any other category name, and an awkward mouthful to boot. So I’d love to hear a better one; if you have such, please share it! In the mean time, I think “HVSP” has merit because:

• The “Processing” part should be noncontroversial.
• “High-Volume” is inherent to the challenge. If RDBMS scale well enough for your use case, using something less powerful is probably silly.* Similarly, while Oracle shines at high-volume OLTP workloads, there are many cheaper DBMS that do a fine job of OLTP at lower volumes.
• “Simple” is the core principle of NoSQL systems, which drop joins and transactions as being too much foofarah. That only makes sense at all under the assumption that you have bone-simple queries and updates, so that programming around the lack of joins and transactions isn’t all that much of a burden.
• Something similar is true of sharded MySQL.
• Less obviously, “simple” is a core principle of relational OLTP as well. The point of the relational model is to cap the complexity of data operations, or more precisely to hide that complexity from programmers.
• And overloading the word “simple” a bit, it’s fair to say that if you’re reading or writing one record at a time, you’re doing something relatively simple, at least as opposed to what you do in analytic processing. The OLTP vs. OLAP distinction is preserved in this name change.
• The whole thing matches my definition above, namely “what happens when lots of people want to get small amounts each of information in or out of a database at the same time.”

*Assuming, of course, that rows-and-tables are a good metaphor for your data structure in the first place.

Systems I’m leaving out of the HVSP and hence also NoSQL categories include:

• Hadoop and other batch-oriented MapReduce. Hadoop isn’t part of NoSQL. I’m pretty sure that Cloudera CEO Mike Olson agrees with me.
• More generally, non-SQL data stores that don’t meet the HVSP criteria. Dave Kellogg stretches things when he claims that MarkLogic is a NoSQL system. (But then, that was in a post where he seemingly praised a train wreck of an article.)

But hey – what good is a categorization if it doesn’t leave some things out?

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

• Facebook invented and is adopting Cassandra.
• Twitter is adopting Cassandra.
• Digg is adopting Cassandra.
• LinkedIn invented and is adopting Voldemort.
• Gee, it seems as if the super-scalable website biz has moved beyond MySQL/Memcached.

But in addition, he provides a lot of useful links, which DBMS-oriented folks such as myself might have previously overlooked. Following those trails gets one to, among other things:

• A September, 2009 post outlining Digg’s reasons for moving to Cassandra. The core idea is that joining two tables is expensive; it’s cheaper to store the results prejoined on disk. Details are provided.
• A February, 2010 post outlining Twitter’s reasons for moving to Cassandra. They boil down to “sufficiently scalable, sufficiently simple, sufficiently robust, robustly open source.”
• A Flickr slide presentation saying “normalization is for wimps”. They seemed to be staying with MySQL, but lusting after XPath.
• A nice Cassandra technical overview by Evan Weaver of Twitter.

I also recall seeing something that said “We have 13X as many queries as updates, so of course we should optimize for reads,” but I can’t find that now. The classical OLTP answer to that would probably be “Yeah, but by the time you’re two-phase-committing and integrity-checking all the part of that update, it turns out updates are still what you should optimize for.” Well, what if the update is so simple that that’s no longer a valid argument?

There certainly seem to be some non-obvious technical choices being made here, with options being conflated that perhaps shouldn’t be. In particular, I wonder whether things are being written to cheap disk in a really fast way when it might be better to keep them in more expensive RAM or, perhaps better yet, solid-state memory. Perhaps then the functionality/performance tradeoff wouldn’t be so painful.

On the other hand, the designers of the world’s most scalable websites — e-commerce sites perhaps excepted — seem pretty unanimous in thinking it’s best to bake some database/integrity management into the applications, rather than offload it all to an RDBMS. Why? Because the transactions are so simple that hand-coding all that isn’t prohibitive. And of course because of their extreme performance and scalability needs.

I’m not sure on what basis one could argue that they’re wrong.

I just don’t see Oracle abandoning HPC entirely. I think it may call it by some other name or describe it differently, but it will be in the high throughput computing business for the foreseeable future.

There are some interesting angles for it to pursue. Many of its best commercial customers have sizeable HPC or HPC-like workloads that Oracle can now (with the addition of Sun) compete for. I don’t see it passing up those opportunities.

Oracle can also look to specialize on certain subsets of the market and provide more of a solution rather than piece parts. I wouldn’t be surprised to hear of it offering an Exadata-like system that is optimized for, say, seismic or financial services. In fact, Exadata as it stands today is a decent fit for financial service analytic workloads.

HPC can be a profitable business and, in a lot of organizations, it’s growing faster than traditional business processing. From Oracle’s perspective, what’s not to like?

Now, except for the Exadata-in-financial-services comment, that’s not directly an argument for the convergence of number crunching and data management.  However, I think Netezza and Aster Data are showing the way for that convergence. So, up to a point, is the scientific-research community. And of course the Hadoop guys think they have the best way to that convergent future.

But if Dan Olds is right that the best technologies for Oracle to pursue HPC and big-data processing with aren’t all that far apart, then the chances that Oracle will indeed pursue their convergence are pretty high. And that would amount to critical mass for the trend.

One of the two posts inspiring me to mention this is a high-level one on “technical debt“, reminding us why things don’t always get done right the first time, and further reminding us that circling back to fix them sooner rather than later is usually wise. The other connects two observations that individually have great merit (at least if you don’t take them to extremes):

• Update-in-place is passe’
• So is elaborate up-front database design

Specific points of interest here include:

• Most data never gets changed after being written. Update-in-place doesn’t save all that much in storage hardware.
• Update-in-place interferes with a lot of modern optimizations in analytic DBMS design.
• Knowing what values data had in the past is interesting in and of itself.
• So, potentially, is knowing what “dirty” data end-users — especially customers and prospects — decided to enter.
• The “right” amount of data validation is application-dependent. For example, if data validation involves torturing your customers, maybe it’s not such a good idea. (Great observation by Chris.)
• If you have the old data as well as the new, the harm of having “bad” updates is lessened. (Central connecting observation by Chris.)
• People enter data inconsistently. MDM (Master Data Management) and data cleansing tools fix much (admittedly not all) of the harm. Computers are cheaper than people. You do the math.
• Data is increasingly being managed in non-relational and/or non-persistent ways. Get used to it.
• As the NoSQL guys point out, some of today’s most demanding applications have extremely simple schemas.

For starters, there’s a lot of new analytic functionality. This isn’t Aster/Netezza-style ambitious. Rather, there’s a lot more SQL-99 functionality, plus some time series extensions of the sort that financial services firms – an important market for Vertica – need and love. Vertica did suggest a couple of these time series extensions are innovative, but I haven’t yet gotten detail about those.

Perhaps even more important, Vertica is cleaning up a lot of its previous SQL optimization and execution weirdnesses. In no particular order, I was told:

• Vertica’s delete performance is up “literally” 30-100X, at least in the case of “large” deletes. Performance for “large” updates has been enhanced as well.
• Vertica has finally cleaned up all vestiges of its prior bias to star schemas. For example, Vertica concedes that its product previously would sometimes force a star execution plan that wasn’t really appropriate.
• It is no longer the case that you need to define projections before you load a table into Vertica. This is now fully automatic.
• Vertica 4.0 automatically redesigns the database when new nodes are added to the system.
• When a database designer does hand-tune projections – and there’s no shame in this still being a possibility in Vertica 4.0 – that hand-tuning is now pulled back into the automatic generation/recommendation/whatever wizards for further projections. I.e., there’s a kind of DBA round-trip engineering going on.
• Vertica used to require that tables being joined be identically “segmented” (I think this means distributed across joins). That is no longer the case in 4.0.
• In connection with this new-found flexibility, Vertica now supports full outer joins directly, rather than requiring the left outer join/right outer join/UNION kluge.
• The Vertica 4.0 optimizer is smarter than its predecessor about things like predicate pushdown into subqueries, or exploiting commonality between predicates and partition keys.
• There’s a fundamental change that I don’t understand very well in the Vertica execution engine basic unit of work. It sounds as if in the past all the disk-based data containers the query needed got opened at once and read into memory, whether or not there was enough RAM and CPU cores to handle them, and this problem has now been fixed.
• Vertica always seemed to say that you could query immediately on new data, because even if it hadn’t hit disk yet – the ROS (Read-Optimized Store) – it was available in memory – the WOS (Write-Optimized Store). And queries were in essence federated between the ROS and WOS. But apparently it’s a new feature in Vertica 4.0 that you can read totally fresh data without locking. I confess to not understanding this very well either. (It has something to do with what  Vertica calls “Epochs”.)
• Temporary tables can now be created in Vertica on a local/session basis without any DDL. Make temporary tables easier and more performant is important for a variety of reasons:
  • Microstrategy, Company V* et al. use lots of temp tables. E.g,, Company V on Vertica has 3000 permanent tables and 5-7000 temporary ones.
  • Vertica rightly points out that temporary tables are also important for ELT (Extract/Load/Transform).
  • Vertica further says that single-node OEMs such as security appliance vendors use lots of temp tables.

*Company V = one of the more prominent vertical-market application providers.

In other Vertica highlights:

• It sounds as if 4.0 is the first Vertica release with what I would regard as serious workload management.
• While Vertica has stored and retrieved Unicode since Vertica 3.5 or so, 4.0 will be the first Vertica release in which Unicode is sorted and collated properly.
• Stored-procedure-like functionality is still a future for Vertica.

• The most important issue in database and analytic technology, in my opinion, isn’t technological at all – rather, it’s the legal and political steps needed to preserve liberty in the face of advancing, intrusive technology.
• Another important issue for society – and this one does involve a lot of technology – is scientific number crunching. In particular, database technology for scientific computing needs to be developed much further. I’ll have more to say on all this soon.
• More generally, technology needs to keep advancing for parallel analytics. Fortunately, it is. Watch this space over the next few weeks.
• Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. I agree.
• Data volumes will keep going up, up, up. Technology needs to keep evolving accordingly. Much of what I write is on that subject.
• Data needs to be processed and analyzed at very different latencies. And there’s much further to go in integrating disparate latencies.
• Analytic database management in the cloud hasn’t been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running (a key subject of discussion at the Boston Big Data Summit).
• Better business intelligence user interfaces are increasingly available. I’m thinking particularly of approaches with buzzwords like visualization/interactive exploration or faceted. But they aren’t well-integrated into the overall analytic stack, as big BI vendors are trailing the smaller ones in this regards. (Part of the problem relates to my previous point.)
• Application development over text search isn’t in the same league as application development over relational DBMS. The choices are mainly XML (e.g., MarkLogic), SQL for text integrated into RDBMS (limited by the weakness of those integrations), and something like Attivio’s Java SDK. There’s a major conceptual barrier in building those apps, namely the unpredictability of query results. Still, it should be possible to do better.
• Similarly, text analytics and conventional analytics exist well side by side. They can even be in the same database and/or dashboard, although in practice that is limited by the strong SaaS focus of text mining vendors and users. But analytic integration of them is really hard. Linguistic imprecision is, in my opinion, only the #2 reason for this difficulty. The #1 reason is that trends detected by text analytics are much less precise than trends on tabular data – e.g., a 50% increase in a certain kind of complaint may be no more significant than a 5% change in a revenue variable.
• I’m increasingly persuaded that graph analytics can be handled without a graph-centric data model. But right now, it isn’t being handled well at all. Lots more needs to be done – although when it is, it will just exacerbate the privacy/liberty dangers that so concern me.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology

One section covered recent/ongoing/near-future trends that I thought were particularly interesting, including:

Simpler database technology, by which I mean DBMS that are:

• Easier to administer than market-leading systems …
• … even if at the cost of being special-purpose
• E.g.,
  • MySQL and older mid-tier RDBMS such as Progress
  • Many analytic DBMS and appliances, most notably Netezza’s

For general purpose or OLTP uses, I’m not a big fan of MySQL (not enough progress in making it industrial-strength), PostgreSQL (no good company behind it – I’m a non-fan of EnterpriseDB), or Ingres (open source or not, it’s an antiquated system that hasn’t been invested in as much as Oracle, DB2 or SQL Server).

But I get the impression there are a lot of contenders among small startups, featuring very new architectures for OLTP or general-purpose database management. VoltDB comes to mind. NimbusDB is finally within range of getting funded. Dan Weinreb told me Friday he knows of a bunch of others as well. And that’s all before we even get into the NoSQL kind of alternative.

Flexible storage architectures. That’s starting out with an emphasis on hybrid columnar, as in the examples of Vertica and Greenplum. Oracle (to whom I’m under no NDA obligation) and other vendors (to whom I am) are going that way as well.

Multi-tier database architectures, by which I mean at least two things:

• The database tier/server tier split of Exadata
• Hybrid RAM/disk architectures, examples of which include
  • Vertica’s RAM-based write-optimized store
  • Sensage’s CEP-in-the-DBMS
  • This in-memory analytics stuff we keep hearing about from the BI vendors
  • Any true in-memory/disk hybrid, such as the regrettably sidelined solidDB
  • Smart thinking by numerous DBMS vendors about optimizing the use of RAM and/or Level 2 cache

Netezza is particularly interesting to watch in this regard because it:

• Had a pretty strict storage/other processing split in prior product generations and …
• … ditched that in its latest generation …
• … which however is focused on optimizing the use of RAM cache

Also noteworthy is Petascan, the stealth-mode –and therefore harder to watch right now – company I keep teasing about, which makes a strong case for carrying the database/storage tier split into the flash/solid-state memory technology generation. Calpont also has a server/storage tier split, but that’s of mainly theoretical interest unless and until Calpont actually ships an MPP version of InfiniDB.

Cheaper parts, which have of course been a huge trend for decades. Solid-state memory will soon conquer the world. Meanwhile, cheaper sensors drive that machine-generated data I keep talking about.

An ever-better understanding of scale-out technology, in several respects, including:

• Query, notably data movement for MPP DBMS
• Update, especially minimalistic DBMS approaches, be they sharded MySQL or more NoSQLish
• Number-crunching, especially via MapReduce and/or parallel analytic libraries integrated into DBMS

Cool trends I touched on more briefly include:

• More data being available for analysis. This was a core theme of my Enzee Universe keynote speeches; there are also some notes on it in my post based on my Boston Big Data Summit talk.
• More users being served by analytics. Ditto.
• Data exploration/visualization, ala QlikView, Spotfire, or Tableau, and also the faceted stuff.
• The democratization of data mining. But I’m not as sure of that one as of the others…

One area I flat-out forgot to mention is easy data mart spin-out.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Open issues in database and analytic technology

• Solid-state memory will soon be the right storage technology for a large fraction of databases, OLTP and analytic alike. I’m not sure whether the initial cutoff in database size is best thought of as terabytes or 10s of terabytes, but it’s in that range. And it will increase over time, for the usual cheaper-parts reasons.
• That doesn’t necessarily mean flash. PCM (Phase-Change Memory) is coming down the pike, with perhaps 100X the durability of flash, in terms of the total number of writes it can tolerate. On the other hand, PCM has issues in the face of heat. More futuristically, IBM is also high on magnetic racetrack memory. IBM likes the term storage-class memory to cover all this — which I find regrettable, since the acronym SCM is way overloaded already.
• Putting a disk controller in front of solid-state memory is really wasteful. It wreaks havoc on I/O rates.
• Generic PCIe interfaces don’t suffice either, in many analytic use cases. Their I/O is better, but still not good enough. (Doing better yet is where Petascan – the stealth-mode company I keep teasing about – comes in.)
• Disk will long be useful for very large databases. Kryder’s Law, about disk capacity, has at least as high an annual improvement as Moore’s Law shows for chip capacity, the disk rotation speed bottleneck notwithstanding. Disk will long be much cheaper than silicon for data storage. And cheaper silicon in sensors will lead to ever more machine-generated data that fills up a lot of disks.
• Disk will long be useful for archiving. Disk is the new tape.

*When the first three people to the question microphone include both Mike Stonebraker and Dave DeWitt, your thinking tends to clarify in a hurry.

Related links

• A slide deck by C. Mohan of IBM similar to the one he presented at the NEDB Summit about storage-class memories.
• A much more detailed IBM presentation on storage-class memories.
• Oracle’s and Teradata’s beliefs about the importance of solid-state memory.
  

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Interesting trends in database and analytic technology
• Open issues in database and analytic technology