• Dwight Merriman of 10gen (MongoDB)
• Damien Katz of Couchio (CouchDB)
• Matt Pfeil of Riptano (Cassandra)
• Todd Lipcon of Cloudera (HBase committer)
• Tony Falco of Basho (Riak)
• John Busch of Schooner
• Ori Herrnstadt of Akiban

By no means do I have time to do these conversations justice, in terms of giving them the write-ups and/or immediate follow-up that they deserve. Indeed, I’ll leave for vacation Saturday morning with my 2000-word NoSQL article still unwritten. So I’ll dump as many observations as I can into one or a few posts now, and play catch-up later as circumstances allow.

In no particular order:

• A number of NoSQL offerings have had more uptake to date than most of the scale-out SQL offerings have.
• “Document-oriented” NoSQL projects CouchDB and MongoDB have probably had the most users get into production, but perhaps for pretty small systems.
• Cassandra and Hbase — the column-group-architecture guys — have probably had the most bang-in-lots-of-writes HVSP production uptake.*
• I didn’t talk customer count with Schooner, but the decently-stocked Schooner customer page suggests Schooner may be something of an exception to these generalities.
• A lot of these companies are in the low-to-mid-teens of employees.
• The SQL-oriented companies, despite having fewer or no customers, often seem to have more money. (One reason I get the impression SQL guys have more money is, frankly, that more of them are talking about engaging my services.)
  • Schooner cites $20 million in VC.
  • Clustrix cites a figure close to that.
  • Basho cites $10 million, plus a new round of $1.5 or $2 or $2.5 million. The new round is at a  lowered valuation.
  • That same site says Tokutek finally was able to raise some VC. Congrats!
• It’s only a two-company trend, but I was pleased to hear that both 10gen/MongoDB and Akiban were seeing Drupal as a major use case or potential use case. No word on rescuing WordPress from its MySQL implementation, alas, but it seems that a Drupal site typically has 40-200+ tables, while a WordPress one has 10ish.
• Another trend I think I’m seeing is serious object-oriented apps banging things straight into a simple back end. Workday is a huge example of that. Akiban hopes to do something similar with Hibernate.
• Stability and maturity are still issues for many of these products. E.g., HBase isn’t even in Release 1.0 yet. Ditto Cassandra, and surely many of the others. Unsurprisingly, making Cassandra stable is still a challenge.

*As is common for terms I suggest, the “HVSP” name is not getting any traction. What do you think of Marton Trencseni’s suggestion of OLRP, for OnLine Request Processing?

One thing that makes following this area interesting is that so many projects are open source, leading there to be a lot of information in the wild. I hardly have time to read the mailing list for each project; but the people I talk with do, and often they may sorta kinda remember something somebody else posted one or several months back. As just one example, the mailing lists are said to confirm:

• Contrary to rumor, Facebook hasn’t moved in-box search off of Cassandra.
• Apparently, however, it’s true that Cassandra inventor Facebook has stopped working on Cassandra, and Facebook’s core Cassandra developers have shifted over to HBase.

Also, figuring out usage of open source software can be … interesting.

• People who use open source software don’t have to reveal themselves, as there’s no purchase transaction to kick things off.
• On the other hand, if they’re serious enough in their use, they often do.
  • There are two main ways to get tech support for open source software — the community or a company that sells support — and both ways let the main support-selling company know that one is a user.
  • Some folks even add themselves to open lists of users, for example these rather long lists for HBase and CouchDB.
  • Or they show up at conferences. For example, two tweets from Riptano founder Jonathan Ellis suggest at least 30 production Cassandra users were represented at a recent event. That’s more detail than his colleague Matt Pfeil wanted to give me when talked.

OK. This post has gotten pretty long, even without me saying anything resembling an overview of any of the seven companies I listed up top, or of their products’ adoption. So I’ll just publish this now, and edit in links below to follow-on posts if and when they become available.

That said, I think I found four sensible points in Pentaho’s Hadoop strategy, namely:

1. If you use an ETL tool like Pentaho’s to move things in and out of HDFS, you may be able to orchestrate two more steps in the ETL process than if you used Hadoop’s native orchestration tools.
2. A lot of what you want to do in MapReduce is things that can be graphically specified in an ETL tool like Pentaho’s. (That would include tokenization or regex.)
3. If you have some really lightweight BI requirements (ad hoc, reporting, or whatever) against HDFS data, you might be content to do it straight against HDFS, rather than moving the data into a real DBMS. If so, BI tools like Pentaho’s might be useful.
4. Somebody might want to use a screwy version of MapReduce, where by “screwy” I mean anything that isn’t Cloudera Enterprise, Aster Data SQL/MapReduce, or some other implementation/distribution with a lot of supporting tools. In that case, they might need all the tools they can get.

The first of those points is, in the grand scheme of things, pretty trivial.

The third one makes sense. While Hadoop’s Hive client means you could roll your own integration with your own favorite BI tool in any case, having somebody certify it for you themselves could be nice. So if Pentaho ships something that works before other vendors do, good on them. (Target date seems to be October.)

The fourth one is kind of sad.

But if there’s any shovel-meet-pony aspect to all this — or indeed a reason for writing this blog post — it would be the second point. If one understands data management, but is in the “Oh no! Hadoop wants me to PROGRAM!” crowd, then being able to specify one’s MapReduce might be a really nice alternative versus having to actually code it.

It also is understood that, realistically, I can’t be expected to know and mention the very latest news for all the many products in the categories. Even so, I think this would be fine time to check just where NoSQL and HVSP adoption stand. Here is most of what I know, or links to same; it would be great if you guys would contribute additional data in the comment thread.

In the NoSQL area: 

• Back in April, the VoltDB guys told me they thought Cassandra and HBase were the two NoSQL systems with the most momentum.
• I know distressingly little about HBase adoption, but a source who may or may not wish to remain anonymous was kind enough to alert me that Twitter and StumbleUpon each have ~30 node deployments, for analytics and analytics/HVSP respectively.
• I wrote in detail on Cassandra adoption last month. News since then includes:
  • Facebook is rumored to have dropped Cassandra completely.
  • Twitter clarified that it may not be quite as lovestruck by Cassandra as before, but they’re still very close friends.
  • It’s not obvious that the Cassandra Summit unveiled a lot of new adoption stories.
• Northscale’s Membase is still in its early days.  Zynga is bought in, however, as is something called NHN Korea. (Edit: I subsequently saw NHN Korea on a prominent SEO expert’s list of the top half dozen or so search engines in the world. Who knew?)
• Basho has listed a few Riak customers. If memory serves (I haven’t spoken with Basho for a while, and some of my notes are misplaced due to some computer sloppiness), Basho has a few dozen customers in total.
• Mozilla has a 4 machine, 64 core Riak cluster in production.
• Hypertable has a few users/project sponsors, Baidu being the biggest name among them.
• I don’t really know how the MongoDB/10gen guys are doing. I think this is at least as much my fault as theirs. Anyhow, they seem to have links to a couple of folks who have written about MongoDB usage.
• NimbusDB is still in stealth mode. I’d be surprised if they had users for a while yet, since in January they didn’t yet sound as if development was very far underway. (Actually, I forget whether NimbusDB is supposed to be SQL-based or not.)

Among the SQL or SQL-friendly guys:

• Clustrix says it has a few production users, some big-name, but is not disclosing them yet.
• dbShards has around 6 customers, including Facebook. (Facebook may outpace even Twitter and Zynga in using the most products mentioned in this post.)
• As of May, VoltDB had one paying customer, plus 150 beta customers who weren’t in production yet.
• Akiban says they’ll get me up to speed on Thursday.
• ScaleDB seems to be pedaling along in perennial beta. Whether ScaleDB has any actual beta users is less clear. On the plus side, checking that out uncovered a pretty funny April Fool blog post.
• Groovy Corporation seems to have disappeared, or morphed into something called uCirrus, or something like that.

To review, Membase is a reasonably elastic persistent data store, sporting the memcached API, making memcached/Membase an attractive alternative to memcached/sharded MySQL. As of now, Membase is a pure key-value store.

Northscale defends pure key-value stores by arguing, in effect: 

• You can do a lot with entity-attribute-value triples.
• If your key looks like an entity-attribute concatenation, then your entity-attribute-value triple can be transformed into a key-value pair.

Northscale has a point. Still, I think that in most use cases you’ll want a data model and/or data access methods that are at least a little richer than pure entity-attribute-value.

Fortunately, that’s the direction Northscale is taking Membase. I don’t get the impression that the details have been worked out yet, but the general idea is:

• Northscale is putting a publish-subscribe interface into Membase it calls “tap,” useful for replication, node rebalancing, etc.
• Tap will also serve to connect Membase data to a Membase feature Northscale calls “Node Code,” which will be code that runs in a separate process on each Membase node.
• Node Code will include things like:
  • Language run-times
  • Standard libraries for things like index-building

Will Membase Node Code be a close substitute for relational DBMS functionality, or even the Cassandra architecture? I doubt it, especially at first. But at least it will keep Membase developers from getting locked in to a very simple and restrictive data management paradigm.

• People
• Financial trades
• Electronic networks
• Everything else

The most varied, interesting, and valuable of those four categories is the first one.

That may change some day, with the growing importance of machine-generated data, and of big-data science in particular. But I think it’s a fair assessment at the present, and for at least the next few years.

Some of the most interesting use cases are concentrated in the areas of identifying individuals, groups of people, or behaviors of (groups of) people. For example:

• comScore works hard to identify individual web surfers – i.e. to deanonymize them — even though they may have given incomplete or false personal information.
• Other companies at least try to figure out which information in a user’s profile is unreliable, so as to classify them better. (Yes, there are 62-year-old video-game-obsessed Lady Gaga fans, but that’s generally not the way to bet.)
• Multiple telecom vendors try to identify who their most influential customers are (to a first approximation, they’re the ones most often called by the most people, but it surely gets more sophisticated than that). This information is then used to reduce churn, either by working hard to retain those users, or – if they do churn – to move very fast to retain the business from their friends.
• Other kinds of companies do similar kinds of analysis, to the extent that they have enough of a social graph to do so. (This application is a case where the term “social graph” is not a misnomer.)
• Turing detectives (I just coined that phrase) try to determine whether users are humans or bots.
• Central to detecting insurance fraud is identifying suspiciously close connections between claimants, service providers, and so on.
• Identifying groups of people is also important in flagging insider trading. Even more important are other kinds of analysis, along the lines of “is this normal innocent trading behavior?”
• Intelligence agencies try to detect networks of terrorists and their sympathizers. They further try to identify unusual patterns of communication or meetings along those networks that might indicate terrorist acts are being planned. (Civilian law enforcement agencies can use similar techniques.)

In most cases, the analysis and/or run-time execution of the relevant models is done with the help of analytic DBMS. Other technologies that come into play include non-DBMS MapReduce (Hadoop), graph engines, and CEP (Complex Event Processing). The vendor most heavily represented on that list is probably Aster Data, because:

• Aster Data is focused on hard-core analytics.
• I talk a lot with Aster Data, and in particular had a long, detailed use-cases discussion with them last week.
• The comScore example happens to come from a speaker at an Aster event I also participated in.

And by the way, all this only scratches the surface of what will be possible down the road. It’s based mainly on where you live, what you purchase, how you behave on websites, and who you communicate with. Other kinds of data, which could be used to be yet more intrusive, generally aren’t involved.

I actually have two points in drawing up this list. One is golly-gee-whiz about how a lot of analytically sophisticated applications are actually getting into production. The other is to highlight the privacy and liberty threats If This Goes On Unchecked (which is why I didn’t include some other less-people-focused examples). There’s also a related danger that, to the extent we don’t get some smart regulations to keep us safe(r), we’ll get a bunch of stupid regulations instead.

The Analytic Era has only just begun.

• Teradata is discontinuing Xkoto’s existing product Gridscale, which Scott characterized as being too OLTP-focused to be a good fit for Teradata. Teradata hopes and expects that existing Xkoto Gridscale customers won’t renew maintenance. (I’m not sure that they’ll even get the option to do so.)
• The point of Teradata’s technology + engineers acquisition of Xkoto is to enhance Teradata’s active-active or multi-active data warehousing capabilities, which it has had in some form for several years.
• In particular, Teradata wants to tie together different products in the Teradata product line. (Note: Those typically all run pretty much the same Teradata database management software, except insofar as they might be on different releases.)
• Scott rattled off all the plausible areas of enhancement, with multiple phrasings – performance, manageability, ease of use, tools, features, etc.
• Teradata plans to have one or two releases based on Xkoto technology in 2011.

Frankly, I’m disappointed at the struggles of clustering efforts such as Xkoto Gridscale or Continuent’s pre-Tungsten products, but if the DBMS vendors meet the same needs themselves, that’s OK too.

The logic behind active-active database implementations actually seems pretty compelling: 

• You may well be keeping a second copy of your database for high availability/hot standby.
• You might even be keeping a third copy for off-site disaster recovery.
• In some cases, you might have reasons beyond disaster recovery to distribute a database around the world.
• So why not allow queries to be run against all the copies?
• And by the way, splitting the workload up a bit by kinds (e.g., long-running vs. short query) might let you optimize the implementation of each copy of the database. (This last point becomes even more important with the rise of solid-state memory.)

Analytic DBMS vendors pretty much all need to offer this. (Possible exception: If they have a data-mart-only positioning so extreme that customers will never care about any form of failover.) That said, I must confess to not having done a good job of tracking who does or doesn’t have which features in this area to date; informative comments to this post in that regard would be much appreciated!

• Despite heavy emphasis on the word “sharding,” dbShards’s scale-out is transparent to the application programmer. E.g., in dbShards + MySQL, the APIs are more or less the same ones you’d expect for MySQL (JDBC, etc.)
• If the DBMS underneath is ACID-compliant (e.g., MySQL + InnoDB), then the dbShards version is ACID-compliant too.
• Beyond those basics, I forgot to check the fine details of dbShards’ MySQL (or PostgreSQL) syntax support. Todd Hoff, however, did not forget.
• dbShards keeps copies of each shard on two different servers, via asynchronous log-shipping. This allows for failover in both planned and unplanned outages.
• dbShards wants you to distribute big tables among shards via a “shard key,” which is a lot like the distribution key in MPP analytic DBMS. You’re encouraged to replicate small, low-update-volume tables across each shard.
• Cory says that dbShards has good join performance when – you guessed it! – everything being joined is co-located shard-by-shard, because the tables were distributed on the same shard key and/or replicated across each shard. Cory can’t imagine why you’d want to do an inner join under any other circumstances.
• The basic dbShards query execution model is: A query comes in; it’s parsed; a shard key is automagically detected (one hopes); the “global configuration file” is checked to see which shard to ship the work off too. I forgot to ask whether lookup was done via a hash table (the obvious guess) or something else. The programmer can put hints in the code comments to direct the sharding, but Cory asserts those aren’t needed very often.
• Cory says that insert performance with dbShards + MySQL + InnoDB is 1500-3000 inserts per shard per second, scaling almost linearly with the number of shards. I forgot to ask how many shards this had been tested for.
• If you want blazing dbShards performance, Cory’s base-case figure is 25 gigabytes of data per node, so that the most commonly used indexes can camp out in memory. (I forgot to ask what kind of hardware he was assuming per node.) This is if you’re going to be doing joins or aggregrations. If it’s just single-row inserts and updates, or if your performance requirements are lower, you can go with 10X that figure.
• Cory tells stories wherein going from an unsharded database to 4 or so shards took database re-indexing time down 50X or more. Apparently, such tasks can be exponential or even super-exponential with database size over InnoDB. (That said, I’d be surprised if all large InnoDB users suffered from that problem to the same degree.)
• dbShards’ customer workloads are all >= 50% reads. This is reflective of dbShards’ design priorities.
• As long as it can be in charge, dbShards is happy to interface to whatever kind of database backup software you want to use on a node by node basis. (dbShards wants to drive your backup software for you so that it can be sure the replicas are handled properly.)
• It’s “fairly common” for dbShards to be paired with memcached. I forgot to ask whether memcached typically lived on its own pool of servers, or on the same pool that runs dbShards.
• Future DBMS options under consideration for dbShards include Oracle and (unspecified) in-memory.

Business highlights for CodeFutures and dbShards include:

• dbShards’ price is $5000/server/year, including support and OEMed MySQL, with stated quantity discounts up to 40%.
• dbShards cloud pricing is different (on a usage basis).
• dbShards has 6 or so customers, half each on-premises and in the cloud. One of them is Facebook. (Those “100s” of customers mentioned on the dbShards website are for a fairly unrelated product.)
• CodeFutures has been at this 2 ½ years or so. There is no venture capital in the company.
• Early deals dbShards deals have evidently involved a fair amount of professional services.
• Counting contractors, Code Futures has 10-12 people, which has been as high as 15.
• Target dbShards customers are as you’d expect. Cory says he’s actually been more successful getting early-adopted money out of Web companies than Wall Street firms.
• There are a couple of dbShards PostgreSQL customers for greenfield applications. Most dbShards customers and prospects, however, are looking to scale out existing apps.
• Despite its connection to open source DBMS, there’s nothing open source about dbShards itself.

• Neil Raden points out that business intelligence dashboards can be dangerously misleading. His reasoning (sound) is that whatever you measure is apt to be distorted by the fact people know they’re being measured. His solution (implied) is to hire a good-looking consultant like himself to do it right.
• I’ve had my issues with Fred Holahan, who was VP of Marketing when I posted that EnterpriseDB was not to be trusted. (That said, Fred is long gone from EnterpriseDB and my opinion hasn’t changed.) But he’s put up a good series of posts on the basis of the open source “progressive engagement” marketing funnel, including this gem on why you shouldn’t count on monetizing your community/free users.
• Oracle plans to increase its acquisition budget. The figure given is $70 billion over the next 5 years. Edit: But see this funny Register followup.
• Clayton Christensen wrote a phenomenal article on how to live a good life, from a very business-y perspective. (Only in one anecdote was it too religiously-oriented for my tastes.) Takeaways include:
  • Your core goals probably revolve around something other than business success. (E.g., family.) Don’t lose sight of that.
  • To the extent you’re a manager or leader, you may have a huge impact on other people’s lives. Use that power in admirable ways.
  • Teach people how to fish for answers, rather than just giving them answers. They’ll probably come to better conclusions than you would have anyway. (This is a core principle in my own consulting.)
  • Take time to reflect. And by the way, the same techniques you use for strategic analysis in business can be applied to your life as well.
• Mark Suster has a pretty good post expanding on my first Christensen takeaway, highlighting a point too often missing from articles in that genre: It’s not just family; it’s also all the cool things around us.
• I haven’t gone through the Hadoop Summit archives yet, but it looks as if there’s a lot of insight there about current Hadoop application activity.
• If you’re a cat lover and don’t hate simple/traditional music, check out Marc Gunn’s cat filksongs, especially the infectious “What Shall We Do With a Catnipped Kitty?” and “Lord of the Pounce”, both playable from the right sidebar of that page (#7 and #10 respectively). Gunn is also a chief perpetrator of the justly (in)famous Do Virgins Taste Better? cycle of filksongs.
• Former SAP exec Dennis Moore offers a theory as to why SAP cares so much about in-memory DBMS. It’s to integrate business processes, because SAP has no other software layer good at doing same. Interestingly, Dennis originated SAP’s previous attempt at meeting a similar need via its composite applications initiative. However, in Dennis’ view this benefit would only be achieved by a major rewrite of SAP’s applications.

• Facebook, which has said it has 150 or so Cassandra nodes (but see below)
• Twitter, which has said it has 45 or so Cassandra nodes
• Rackspace, which used to be Jonathan Ellis’ employer, and now is backing Cassandra company Riptano
• Digg, which along with Twitter and Rackspace was one of the three major users helping advance the Cassandra project
• OpenX, Simple Geo, Digital Reasoning, who Jonathan cited as production users in March
• Cloudkick, as noted and linked in my other post
• Two customers Riptano named at launch (but I’ve forgotten who they were*)

Fetlife, Meebo, and others seem to at least have a healthy interest in Cassandra, based on their level of involvement in a forthcoming Cassandra Summit. That said, the @Fetlife tweetstream features numerous yelps of pain, and I don’t mean the recreational kind. 

*And I can’t easily find a launch press release, whether on the rather minimalist Riptano website or elsewhere.

Beyond that, when Riptano launched in May, the Riptano guys (mainly Jonathan Ellis) said:

• They were sure there were dozens of Cassandra user organizations, maybe even >100. But there weren’t 100s.
• Maybe 20-40% of those Cassandra sites were in production. (But I don’t think I’d multiply that out to suggest there were, say, 35-50 production Cassandra users.)
• 4000 people were going daily to the Apache Cassandra site.
• There were 250 Cassandra downloads daily.
• Lots of startups were using Cassandra.
• Lots of other companies were looking at switching over to Cassandra.
• Many potential Cassandra users had been waiting for a Cassandra company to be available to support it.
• The median number of Cassandra (production?) nodes is probably 8-10. 4 would be a low end figure.

That’s a lot of adoption for a not-even-Release-1 open source project. Even so, there’s a feeling going around that Cassandra has lost some momentum the past couple of months. Most notably, Facebook, which created Cassandra in the first place, isn’t using it for new projects. True, I’m hearing even less evidence that any one of Membase, Voldemort, VoltDB, Akiban, Clustrix, or Riak – for example – is setting the world on fire than I am for Cassandra. But the viable Cassandra alternatives are piling up. Cassandra isn’t the only or even primary game in town, and for that matter I haven’t heard any concise description of a niche in which Cassandra is the unquestioned leader.

Edit: A/the Facebook project that continues to run on Cassandra is Inbox search.

As for Riptano itself:

• Riptano launched with two founders and immediately made an offer to a third guy. I don’t know how many folks they have now, two months later.
• Rackspace put some funding into Riptano.
• Riptano’s strategy sounds a lot like Cloudera’s, by which I mean:
  • Riptano’s business is all services, whether training, consulting, or support.
  • Riptano’s intended main business is obviously support.
  • Notwithstanding the above, Riptano intends to eventually offer proprietary software, bundled with its support services.
  • The first area of focus for that proprietary software is intended to be management tools.
  • I wouldn’t be surprised if, like Cloudera, Riptano tweaks its software focus from “stuff that lets us support you better” to “integration with stuff you pay for.” Those strategies are actually pretty similar.

Riptano seems to be starting out with support pricing around $1,000-$4,000/server/year, before quantity discounts.

Jonathan’s core claims for Cassandra include:

• Cassandra is shared-nothing.
• Cassandra has good approaches to replication and partitioning, right out of the box.
• In particular, Cassandra is good for use cases that distribute a database around the world and want to access it at “local” latencies. (Indeed, Jonathan asserts that non-local replication is a significant non-big-data Cassandra use case.)
• Cassandra’s scale-out is application-transparent, unlike sharded MySQL’s.
• Cassandra is fast at both appends and range queries, which would be hard to accomplish in a pure key-value store.

In general, Jonathan positions Cassandra as being best-suited to handle a small number of operations at high volume, throughput, and speed. The rest of what you do, as far as he’s concerned, may well belong in a more traditional SQL DBMS. 

Further highlights of our talks included, as best I understood them:

• Cassandra is based in parts both on Google’s BigTable paper of 2006 and Amazon’s Dynamo paper of 2007.
  • The core of what Cassandra takes from BigTable is based on log-structured merge trees, which actually entered the computer science literature in 1996.
  • Cassandra’s approach to horizontal scaling, replication, failover, etc. seems to be based Dynamo.
• There seems to be a logical concept of “row” in Cassandra, or it’s at least meaningful to use the SQL/relational concept of a “row” when talking about Cassandra data. However, Cassandra is closer to being a column-based data store than a row-based one. (Not the same thing, but closer.)
• Even so, it only takes a single seek to return a whole Cassandra “row”.
• Cassandra writes data quite differently from the way a classical OLTP DBMS would.
  • Cassandra writes just the data elements – i.e., fields – that are actually being inserted or changed, not whole rows.
  • One benefit is that Cassandra data is very sparse. NULLs aren’t stored in any way, and hence in particular take up no space.
  • Another benefit – and one of the core concepts of Cassandra – is that you can implicitly assume different schemas for different rows of the same “table.” In particular, you can add data for columns that you didn’t envision when you first started storing “rows” of the same “table.”
  • Writes are collected into sorted “memtables,” which from time to time are sent to disk. Once data gets to disk, it’s immutable, except for occasional merge/reorganization/garbage collection.
    • Jonathan claims, plausibly, that this makes write throughput very fast (because the I/O is fundamentally sequential in nature.)
    • The default as to how long data typically stays in memory before it gets persisted to disk is “whichever comes first of {64 MB written, 300k updates, 1 hour}”.
  • Cassandra has durability – guaranteed non-loss of data – assuming fsync is turned on. fsync seems to create a 15% or so overhead.
  • However, Cassandra has no concept of a “transaction.”
  • As one would expect, data can be read even before it has been persisted to disk.
• According to Jonathan, Cassandra can do about 14,000 writes or 7,000 reads per second, on a quad-core server.
  • Those figures scale pretty linearly with the number of servers. (There’s some overhead for network latency.)
  • Those figures assume a five-column row.
  • Cassandra’s write-performance figures are only “mildly sensitive” to the width of the row. E.g., doubling row width only gives a 15-20% throughput hit, due to some fixed per-row overhead. That said, I imagine going 100X in row width would create a major slowdown, although perhaps while measuring width more in bytes than in column count.
  • Cassandra’s performance has been growing nicely in each point release. Jonathan thinks this general trend will continue.
• Jonathan thinks Cassandra is pretty good at keeping your data safe.
  • Each node has a commit log.
  • When a node goes down, its writes are buffered until it comes back up.
• You can run Hadoop MapReduce straight against Cassandra files.
• A Cassandra node might hold anything from 10s of gigabytes to multiple terabytes of data. You might want to go with the low end if you want to have lots of cache hits.
• Solid-state storage would speed up Cassandra reads, not writes, and is not widely used with Cassandra yet.
• Jonathan says Cassandra is really good at handling time series data, by which I suspect he means log files. Cloudkick is a user of this capability.

I certainly didn’t grasp everything about Cassandra replication and partitioning strategies. That wasn’t the focus of our talks, and anyway I got the impression they are so flexible that there’s little that can firmly be said about them. But I did get the impressions:

• You set your consistency rules in the Cassandra API, not on a per-table basis. (I think this means that a lack of administrative tools is supposedly a feature, not a drawback.)
• As a practical matter, Cassandra users commonly take one of two approaches to consistency:
  • RYW consistency, most commonly with N = 3 and R = W = 2.
  • Geographically dispersed eventual consistency.
• Cassandra data is most commonly distributed via consistent hashing, but other options are “pluggable.”
• If you add a node, the busiest note automagically decides to ship some data over, reducing its load. Of course, this only works if you get the new node on before the old node is so maxed out it doesn’t have time to do the shipping.

When we talked in March, the next release of Cassandra was going to be 0.7. Cassandra 0.7 was going to be a performance/scalability release, for example fixing the flaw that garbage collection read rows into memory one at a time. After that, Cassandra 0.8 was to be a feature release, with one planned feature being more automatic index management and/or materialized-view-like capability, so as to reduce the burden on Cassandra developers of schema management.

Related links

• My March NoSQL links post included the Google and Amazon papers
• The March 2, 2010 Cloudkick post also linked above goes into a lot of detail, including what they think is great about Cassandra and what they think is still missing