• Facebook et al. are in effect Software as a Service (SaaS) vendors, not enterprise technology users. In particular:
  • They have the technical chops to rewrite their code as  needed.
  • Unlike packaged software vendors, they’re not answerable to anybody for keeping legacy code alive after a rewrite. That makes migration a lot easier.
  • If they want to write different parts of their system on different technical underpinnings, nobody can stop them. For example …
  • …  Facebook innovated Cassandra, and is now heavily committed to HBase.
• It makes little sense to talk of Facebook’s use of “MySQL.” Better to talk of Facebook’s use of “MySQL +  memcached  + non-transparent sharding.” That said:
  • It’s hard to see why somebody today would use MySQL +  memcached  + non-transparent sharding for a new project. At least one of Couchbase or transparently-sharded MySQL is very likely a superior alternative. Other alternatives might be better yet.
  • As noted above in the example of Facebook, the many major web businesses that are using MySQL +  memcached  + non-transparent sharding for existing projects can be presumed able to migrate away from that stack as the need arises.

Continuing with that discussion of DBMS alternatives:

• If you just want to write to the memcached API anyway, why not go with Couchbase?
• If you want to go relational, why not go with MySQL? There are many alternatives for scaling or accelerating MySQL — dbShards, Schooner, Akiban, Tokutek, ScaleBase, ScaleDB, Clustrix, and Xeround come to mind quickly, so there’s a great chance that one or more will fit your use case. (And if you don’t get the choice of MySQL flavor right the first time, porting to another one shouldn’t be all THAT awful.)
• If you really, really want to go in-memory, and don’t mind writing Java stored procedures, and don’t need to do the kinds of joins it isn’t good at, but do need to do the kinds of joins it is, VoltDB could indeed be a good alternative.

And while we’re at it — going schema-free often makes a whole lot of sense. I need to write much more about the point, but for now let’s just say that I look favorably on the Big Four schema-free/NoSQL options of MongoDB, Couchbase, HBase, and Cassandra.

• Dwight Merriman of 10gen (MongoDB)
• Damien Katz of Couchio (CouchDB)
• Matt Pfeil of Riptano (Cassandra)
• Todd Lipcon of Cloudera (HBase committer)
• Tony Falco of Basho (Riak)
• John Busch of Schooner
• Ori Herrnstadt of Akiban

By no means do I have time to do these conversations justice, in terms of giving them the write-ups and/or immediate follow-up that they deserve. Indeed, I’ll leave for vacation Saturday morning with my 2000-word NoSQL article still unwritten. So I’ll dump as many observations as I can into one or a few posts now, and play catch-up later as circumstances allow.

In no particular order:

• A number of NoSQL offerings have had more uptake to date than most of the scale-out SQL offerings have.
• “Document-oriented” NoSQL projects CouchDB and MongoDB have probably had the most users get into production, but perhaps for pretty small systems.
• Cassandra and Hbase — the column-group-architecture guys — have probably had the most bang-in-lots-of-writes HVSP production uptake.*
• I didn’t talk customer count with Schooner, but the decently-stocked Schooner customer page suggests Schooner may be something of an exception to these generalities.
• A lot of these companies are in the low-to-mid-teens of employees.
• The SQL-oriented companies, despite having fewer or no customers, often seem to have more money. (One reason I get the impression SQL guys have more money is, frankly, that more of them are talking about engaging my services.)
  • Schooner cites $20 million in VC.
  • Clustrix cites a figure close to that.
  • Basho cites $10 million, plus a new round of $1.5 or $2 or $2.5 million. The new round is at a lowered valuation.
  • That same site says Tokutek finally was able to raise some VC. Congrats!
• It’s only a two-company trend, but I was pleased to hear that both 10gen/MongoDB and Akiban were seeing Drupal as a major use case or potential use case. No word on rescuing WordPress from its MySQL implementation, alas, but it seems that a Drupal site typically has 40-200+ tables, while a WordPress one has 10ish.
• Another trend I think I’m seeing is serious object-oriented apps banging things straight into a simple back end. Workday is a huge example of that. Akiban hopes to do something similar with Hibernate.
• Stability and maturity are still issues for many of these products. E.g., HBase isn’t even in Release 1.0 yet. Ditto Cassandra, and surely many of the others. Unsurprisingly, making Cassandra stable is still a challenge.

*As is common for terms I suggest, the “HVSP” name is not getting any traction. What do you think of Marton Trencseni’s suggestion of OLRP, for OnLine Request Processing?

One thing that makes following this area interesting is that so many projects are open source, leading there to be a lot of information in the wild. I hardly have time to read the mailing list for each project; but the people I talk with do, and often they may sorta kinda remember something somebody else posted one or several months back. As just one example, the mailing lists are said to confirm:

• Contrary to rumor, Facebook hasn’t moved in-box search off of Cassandra.
• Apparently, however, it’s true that Cassandra inventor Facebook has stopped working on Cassandra, and Facebook’s core Cassandra developers have shifted over to HBase.

Also, figuring out usage of open source software can be … interesting.

• People who use open source software don’t have to reveal themselves, as there’s no purchase transaction to kick things off.
• On the other hand, if they’re serious enough in their use, they often do.
  • There are two main ways to get tech support for open source software — the community or a company that sells support — and both ways let the main support-selling company know that one is a user.
  • Some folks even add themselves to open lists of users, for example these rather long lists for HBase and CouchDB.
  • Or they show up at conferences. For example, two tweets from Riptano founder Jonathan Ellis suggest at least 30 production Cassandra users were represented at a recent event. That’s more detail than his colleague Matt Pfeil wanted to give me when talked.

OK. This post has gotten pretty long, even without me saying anything resembling an overview of any of the seven companies I listed up top, or of their products’ adoption. So I’ll just publish this now.

It also is understood that, realistically, I can’t be expected to know and mention the very latest news for all the many products in the categories. Even so, I think this would be fine time to check just where NoSQL and HVSP adoption stand. Here is most of what I know, or links to same; it would be great if you guys would contribute additional data in the comment thread.

In the NoSQL area: 

• Back in April, the VoltDB guys told me they thought Cassandra and HBase were the two NoSQL systems with the most momentum.
• I know distressingly little about HBase adoption, but a source who may or may not wish to remain anonymous was kind enough to alert me that Twitter and StumbleUpon each have ~30 node deployments, for analytics and analytics/HVSP respectively.
• I wrote in detail on Cassandra adoption last month. News since then includes:
  • Facebook is rumored to have dropped Cassandra completely.
  • Twitter clarified that it may not be quite as lovestruck by Cassandra as before, but they’re still very close friends.
  • It’s not obvious that the Cassandra Summit unveiled a lot of new adoption stories.
• Northscale’s Membase is still in its early days.  Zynga is bought in, however, as is something called NHN Korea. (Edit: I subsequently saw NHN Korea on a prominent SEO expert’s list of the top half dozen or so search engines in the world. Who knew?)
• Basho has listed a few Riak customers. If memory serves (I haven’t spoken with Basho for a while, and some of my notes are misplaced due to some computer sloppiness), Basho has a few dozen customers in total.
• Mozilla has a 4 machine, 64 core Riak cluster in production.
• Hypertable has a few users/project sponsors, Baidu being the biggest name among them.
• I don’t really know how the MongoDB/10gen guys are doing. I think this is at least as much my fault as theirs. Anyhow, they seem to have links to a couple of folks who have written about MongoDB usage.
• NimbusDB is still in stealth mode. I’d be surprised if they had users for a while yet, since in January they didn’t yet sound as if development was very far underway. (Actually, I forget whether NimbusDB is supposed to be SQL-based or not.)

Among the SQL or SQL-friendly guys:

• Clustrix says it has a few production users, some big-name, but is not disclosing them yet.
• dbShards has around 6 customers, including Facebook. (Facebook may outpace even Twitter and Zynga in using the most products mentioned in this post.)
• As of May, VoltDB had one paying customer, plus 150 beta customers who weren’t in production yet.
• Akiban says they’ll get me up to speed on Thursday.
• ScaleDB seems to be pedaling along in perennial beta. Whether ScaleDB has any actual beta users is less clear. On the plus side, checking that out uncovered a pretty funny April Fool blog post.
• Groovy Corporation seems to have disappeared, or morphed into something called uCirrus, or something like that.

• Nothing in my original short post about Clustrix was actually incorrect.
• Clustrix plans to reveal actual production “name-brand” customers soon.
• The name of Clustrix’s software, or at least the guts thereof, is Sierra.
• Clustrix’s products have actually been in general availability since last quarter, with some versions at customer sites for 2 years. Development started 3 ½ years ago.
• Clustrix says its technology is for OLTP systems, which it calls “non-batch/non-analytic,” with mixed read/write workloads. All Clustrix’s example target markets are “internet verticals,” such as photo sharing, gaming, social media, e-commerce, etc.
• Clustrix’s heart is in SQL, as is most of its customer base. Clustrix Sierra’s key-value-store option has little or no performance advantage over Clustrix Sierra’s SQL option, nor any other advantage over SQL that came up in discussion.
• Clustrix Sierra is “wire-compatible” with MySQL, but doesn’t use MySQL code; Clustrix wrote all the code itself.
• Clustrix asserts that Clustrix Sierra supports the “vast majority” of MySQL features. Examples of MySQL features Clustrix doesn’t support at this time are full-text search and geospatial indexing.
• Indeed, Clustrix claims Clustrix Sierra can be used to replace MySQL with few or zero changes to existing applications.
• I specifically asked about referential integrity, which has a poor performance reputation in MySQL. Besides saying they supported it, Clustrix said that some customers actually use referential integrity in some of their less active tables.
• Clustrix Sierra is fully ACID-compliant, with no eventual consistency or RYW consistency story. The default number of copies of each datum is two, and they’re kept consistent via two-phase commit.
• Clustrix Sierra is fully parallel, with no “head” node. I forgot to ask how it was determined which queries would be addressed to and/or controlled by which nodes, but I presume there’s some sort of a load-balancing scheme.
• Clustrix says that because Clustrix Sierra uses MVCC (Multi-Version Concurrency Control), and thus reads and writes don’t block each other, global locks aren’t a major issue. (They’re rare or short or something – I have trouble seeing why they would be non-existent.)
• Clustrix says there’s a second class of locks and latches that are purely local and short-lived, for B-tree indexes and the like. (I didn’t drill down into those either.) I guess this means Clustrix Sierra is B-tree-centric, which makes sense for an OLTP-oriented system.
• Clustrix Sierra distributes data among nodes via consistent hashing (default), range partitioning, or “full distribution”(i.e., copying a – presumably small – table to each node). The choice of distribution plans is manual now; more automation is a future feature.
• Clustrix Sierra’s CBO (Cost-Based Optimizer) is, as one would hope, distribution-aware.
• Clustrix Sierra compiles query fragments and ships them off to the relevant nodes. A fragment might contain both instructions for SQL to be executed locally and for where data is to be sent next.
• Clustrix says that Clustrix Sierra does data migration and redistribution (e.g., when you add a node) transparently online, and further says that in practice this doesn’t cause a performance hit.
• As for Clustrix hardware:
  • Clustrix makes Type I appliances.
  • A Clustrix node contains 2 quad-core chips, 32 gigs of RAM, and 7 160 GB solid-state drives.
  • Specifically, Clustrix is using Intel SSDs, with a SAS interface.
  • Clustrix says solid-state memory isn’t really essential to the product design; it’s just cheap in terms of $/IOPS (I/O Per Second).
• A minimum Clustrix configuration is 3 nodes, for redundancy. After that you can add nodes one at a time. Clustrix says it built a 20-node system in-house, leading me to suspect that customers don’t have anything bigger than 20 nodes either.
• That 20-node Clustrix system was tested to show near-linear scalability. (In discussing this, Clustrix tends to forget to use the word “near”.)
• Clustrix has partnered with somebody to provide global 4-hour-response support. As of now Clustrix seems to be active mainly in North America and Europe.
• Clustrix is formed from the combination of two startups, which I’ve heard elsewhere were called Clustrix and Sprout. Exactly when the combination happened sounds a little different depending on who’s telling the story (one version has the predecessors still being separate well into 2008, but Clustrix implies the combination happened pretty much on Day 1).

• Clustrix is making OLTP DBMS.
• The core problem Clustrix tries to solve is scale-out, without necessarily giving up SQL. (I couldn’t immediately tell whether Clustrix supports NoSQL-style key-value interfaces enthusiastically, grudgingly, or not at all.)
• Unlike Akiban or VoltDB, Clustrix makes database appliances. The Clustrix software seems to assume a Clustrix appliance.
• A key feature of Clustrix’s database appliances is that they rely on solid-state memory. I’m guessing that Clustrix appliances don’t even have disks, or that if they do the disks store some software or something, not actual data. (As previously noted, I agree with Oracle in thinking that much of the progress in database technology this decade will come from proper design for solid-state memory.)
• Clustrix talks of things that sound like compiled queries and attempts to avoid locks. However, it doesn’t sound as extreme in these regards as VoltDB.
• Clustrix also talks of things that sound like consistent hashing.
• The brand name “Sierra” also shows up along with the brand name “Clustrix.”