Most of what’s confusing about the MarkLogic Hadoop Connector lies in two pairs of options it presents you:

• Hadoop can talk XQuery to MarkLogic. But alternatively, Hadoop can use a long-established simple(r) Java API for streaming documents into or out of a MarkLogic database.
• Hadoop can make requests to MarkLogic in MarkLogic’s normal mode of operation, namely to address any node in the MarkLogic cluster, which then serves as a “head” node for the duration of that particular request. But alternatively, Hadoop can use a long-standing MarkLogic option to circumvent the whole DBMS cluster and only talk to one specific MarkLogic node.

Otherwise, the whole thing is just what you would think:

• Hadoop can read from and write to MarkLogic, in parallel at both ends.
• If Hadoop is just writing to MarkLogic, there’s a good chance the process is properly called “ETL.”
• If Hadoop is reading a lot from MarkLogic, there’s a good chance the process is properly called “batch analytics.”

MarkLogic said that it wrote this Hadoop connector itself.

When I realized MarkLogic was claiming the ability to seamlessly integrate short-request and batch analytic processing, I asked about workload management. I gathered that:

• MarkLogic believes that MarkLogic 5 does a great job of granular workload monitoring.
• However, MarkLogic doesn’t have a strong workload management administrative interface. Rather, you may have to do workload management programmatically.

Overall, I think the MarkLogic Hadoop connector could prove pretty useful. The first question I ask somebody who wants to process relational data in Hadoop is “Why not just an analytic RDBMS?” But the natural use cases for MarkLogic are often ones in which you might as well do your analytics in Hadoop, including a 4 billion Word/PDF/image document insurance-industry example I recently encountered, and for which I favor MarkLogic over MongoDB or straight Hadoop alike.

Views of the NoSQL market horse race are reasonably consistent, with perhaps some elements of “Where you stand depends upon where you sit.”

• As James tells it, NoSQL is simply a three-horse race between Couchbase, MongoDB, and Cassandra.
• Max would include HBase on the list.
• Further, Max pointed out that metrics such as job listings suggest MongoDB has the most development activity, and Couchbase/Membase/CouchDB perhaps have less.
• The Cloudera guys remarked on some serious HBase adopters.*
• Everybody I spoke with agreed that Riak had little current market presence, although some Basho guys could surely be found who’d disagree.

*I hope to do a separate post on HBase adoption soon. In connection with that, any info on HBase adoption by Facebook (said to be very heavy), Twitter, et al. would be much appreciated.

The reasons for using NoSQL of course are, in some order, dynamic schemas, scale-out, and open source. I find the scale-out argument somewhat bogus,* but the data model one is very real. Depending on whom you talk with, the most important point about dynamic schemas may actually be that they’re changeable, or it may just be that you don’t have to specify a schema at the time of initial application design. MongoDB gets particular praise as a good platform on which to throw something together quickly, although predictions as to how far the application will then scale may differ depending on whether you’re talking with, say, Max or Todd.

*It’s fair to say that NoSQL systems are more proven in scale-out than most relational DBMS. Even so, I would cringe at any line of reasoning that concluded one should adopt NoSQL because it is more mature than relational alternatives.

Finally, I was perhaps too extreme when I suggested there was no good reason for Oracle to have adopted the major key/minor key approach it took in its NoSQL offering. Todd offered a reason why that approach – which he characterized as similar to Project Voldemort’s – could make sense:

• If you have some kind of global secondary index, it’s hard to maintain that index consistently without what amounts to distributed transactions.
• If you want to avoid the overhead of those, one alternative is a column-group system such as HBase or Cassandra. Those have no indexes at all, except in the sense that a column is its own index.
• Another alternative is to load as much indexing information as you can into the key of a key-value store.

I’d be interested to learn about the Couchbase and MongoDB answers to that challenge.

My favorite relational OLTP scale-out choice these days is the SchoonerSQL/dbShards partnership. Schooner Information Technology (SchoonerSQL) and Code Futures (dbShards) are young, small companies, but I’m not too concerned about that, because the APIs they want you to write to are just MySQL’s. The main scenarios in which I can see them failing are ones in which they are competitively leapfrogged, either by other small competitors – e.g. ScaleBase, Akiban, TokuDB, or ScaleDB — or by Oracle/MySQL itself. While that could suck for my clients Schooner and Code Futures, it would still provide users relying on MySQL scale-out with one or more good product alternatives.

Relying on non-MySQL NewSQL startups, by way of contrast, would leave me somewhat more concerned. (However, if their code is open sourced. you have at least some vendor-failure protection.) And big-vendor scale-out offerings, such as Oracle RAC or DB2 pureScale, may be more complex to deploy and administer than the MySQL and NewSQL alternatives.

The short form of the SchoonerSQL (as Schooner’s product is now called) story goes roughly like this:

• SchoonerSQL replicates data — synchronously if the replication target is local, asynchronously if it is remote.
• Local synchronous replication provides high availability; remote asynchronous replication provides disaster recovery.
• SchoonerSQL’s local synchronous replication also provides read scale-out.
• Schooner has a partnership with Code Futures/dbShards to provide write scale-out via transparent sharding.
• SchoonerSQL has some secret sauce in replication performance. This has the effect of significantly increasing write performance (assuming you were going to replicate anyway), because otherwise you might have to slow down the master server’s write performance so that the slaves can keep up with it.
• Schooner believes it still has some single-server performance advantages as well.

Just to be clear here: Schooner is my client. Code Futures is my client. I introduced them and suggested their partnership. I then introduced them both to a user client, who was sufficiently impressed to seriously evaluate both of them. Even so, some of the Schooner replication story only became clear to me when I visited last Friday.

To flesh that out a bit more:

• Schooner has been making performance tweaks to the MySQL/InnoDB stack for several years. Some of them are still relevant, and can offer 50-100% performance improvements, especially but not only if you use solid-state storage.
• Schooner has found a way to scale up the slave side of master-slave replication, both in the synchronous and asynchronous cases.
  • The core idea of SchoonerSQL parallel (i.e. scale-up) replication is straightforward. The replication log streams in. A chunk is sent off to a CPU core. The next chunk is examined for conflicts with the first chunk. If there are none, it is sent off to a different core to be processed
  • Thus, you can have both local replication/high availability and remote replication/disaster recovery without slowing down writes on the master the way you may have to with other alternatives.
  • Schooner believes this feature alone can make SchoonerSQL 3X faster than MySQL alternatives, at least for writes.

At least in casual conversation, Schooner synthesizes its 1.5-2X single-server figure and up-to-3X clustering figure into a single claim of frequent 2-5X speedup over generic MySQL/InnoDB. But the usual caveats about vendor-supplied performance numbers of course apply.

Finally, some housekeeping:

• At Oracle’s polite request,* Schooner changed its product name to not mention MySQL; hence the moniker SchoonerSQL.
• SchoonerSQL is being launched Monday.
• Schooner has determined that if its version numbers are different from MySQL’s, confusion ensues. So the first ever version under the name SchoonerSQL is SchoonerSQL 5.1, a factoid that Schooner is wisely omitting from the official product launch press release.
• The synchronous part of SchoonerSQL’s replication story dates back to last January’s product release.
• Most of the asynchronous part of SchoonerSQL story is new for Monday.
• dbShards/SchoonerSQL partnership engineering is still underway.

*I mean that non-facetiously. Schooner’s MySQL OEM contract was such that Oracle didn’t have a legal hammer to force the change. Edit: Whoops! There turn out to have been inaccuracies in the original version of this footnote, which I now regret writing. The contract isn’t exactly OEM, and there actually were some trademark-based legal hammers.

• 100s of gigabytes of data at first, growing to >1 terabyte over time.
• High peak loads.
• Public cloud portability (but they have private data centers they can use today).
• Simple database design — not a lot of tables, not a lot of columns, not a lot of joins, and everything can be distributed on the same customer_ID key.
• Stream the data to a data warehouse, that will grow to a few terabytes. (Keeping only one year of OLTP data online actually makes sense in this application, but of course everything should go into the DW.)

So I’m leaning to saying:  

• They should go with a scalable, MySQL-based solution.
  • Lots of third-party software works with MySQL, in case that’s helpful.
  • Yes, any one vendor is small and not yet firmly established, but there are numerous vendors around with interesting MySQL scaling stories.
  • In a vendor emergency, just going with Oracle’s MySQL stuff would probably work …
  • … especially because there are these lovely things in the world called solid-state drives.
  • There’s also good escapability if one wants to move away from MySQL, because everybody knows how to handle MySQL data.
• The first product to look at is dbShards, because it meets all the topology needs:
  • Local scale-out (transparent sharding).
  • Local high availability.
  • Remote disaster recovery (details of that are underway).
• The first analytic DBMS to look at is Infobright.
  • Yes, I know Infobright is focused more on machine-generated data these days, but this client’s analytic needs are so straightforward Infobright should pass with flying colors.
  • The MySQL-to-MySQL aspect should make ETL dead simple.
  • Again, there’s escapability.

Mainly, this is all fine. But I’m getting pushback on the solid-state aspect, for fear that it will compromise public cloud portability.

Am I missing something here? As far as I’m concerned, if you’re planning an OLTP system with a many-year lifespan today, of course you should assume solid-state storage. Maybe you scale out just as far as you would with disk, striping indexes or entire databases across the RAM of multiple servers. It that case, having solid-state backing reduces the risk of bottlenecks. Maybe you don’t scale out as far as you would with disk. In that case, solid-state backing saves you money.

As for public-cloud support for solid-state storage, that’s coming fast, right? (Actually, I have data points in support of that theory, but they’re a bit tenuous.) A large fraction of web businesses with private data centers seem to be using solid-state storage — from Facebook on down — or so the NoSQL/NewSQL/short-request DBMS guys tell me. Surely a number of public cloud vendors are close behind.

memcached is a way to cache data in RAM across a cluster of servers and have it all look logically like a single memory pool, extremely popular among large internet companies. The Membase product — which is what Couchbase has been selling this year — adds persistence to memcached, an obvious improvement on requiring application developers to write both to memcached and to non-transparently-sharded MySQL. The main technical points in adding persistence seem to have been:

• A persistent backing store (duh), namely SQLite.
• A change to the hashing algorithm, to avoid losing data when the cluster configuration is changed.

Couchbase is essentially Membase improved by integrating CouchDB into it, with the main changes being:

• Changing the backing store to CouchDB (duh). This will be in the first Couchbase release.
• Adding cross data center replication on CouchDB’s consistency model. This will not, I believe, be in the first Couchbase release.
• Offering CouchDB’s programming and query interfaces as an option. So far as I can tell, this will be implemented straightforwardly in the first Couchbase release, with elegance planned for later down the road.

Let’s drill down a bit into Membase/Couchbase clustering and consistency.

• When data is written to RAM in memcached, it immediately gets copied to another server. The same is of course true in Membase/Couchbase. The terminology on all this is confusing, but I think:
  • The portion of data that is stored as a primary copy on any given server is called a “shard”.
  • That would seem to make sense, as that data could correspond to what goes — non-transparently — into an instance of MySQL in a classical memcached/MySQL set-up.
• Updates are of course also banged to disk ASAP — but at times of heavy load, that can take a while. A few seconds to a couple of minutes is normal operation; if it takes an hour, you really should buy more hardware. (Or solid-state storage.)
• Similarly, the replication of data to a second machine’s RAM may not happen at times of heavy load — and that’s another sign you don’t have enough machines.
• Each Membase/Couchbase “shard” has lots of logical sub-shards.* (1024 for now, at least as default, although Dustin finds that number excessive and is looking to lower it.)  So if you add a node, some of the sub-shards get sent over to the new node. Unlike the case for straight memcached, no data is lost from cache (and of course not also from the persistent store). Blocking of operations from such a move only happens in narrow time windows, and then only in edge cases.

*Edit: They’re called vbuckets.

So if we consider Membase technology alone, Couchbase is CA in the CAP Theorem.  CouchDB, however, is gloriously AP in the CAP Theorem, in that it was written to assume an occasionally connected topology.* Based on that, Couchbase will allow AP operation between data centers (i.e. “stay synchronized if you can, to within the limitations of physics and so on, but don’t beat yourself up on the rare occasions that you can’t.”) I don’t know that that capability will quite be in the first release of Couchbase, but it’s coming soon.

*CouchDB also has other features friendly to occasionally-connected use cases, such as a lot of flexibility as to which parts of the database are or aren’t synced when you do reconnect. These are at the heart of the Couchbase Mobile offering.

memcached and Membase have a very simple key-value interface. CouchDB adds secondary indexes and so on. I think in the first release of Couchbase this is pretty much like having two different APIs for the same product; more elegant integration is planned down the road, and more language support as well.

The highest-performing way to use Couchbase will probably always be to just pretend it is Membase, which is to say memcached+. Dustin told me of Membase users who demanded 10-40 millisecond response times, and that not even for single queries but rather for sequences of several queries in succession. He further told me of customers asking for 1-200 microsecond response, and insisting on no worse than 1 millisecond. Frankly, the first requirement could be met by lots of technologies I can think of, at least if  you don’t rely on disk; the second is thoroughly impossible if you rely on disk, and pretty demanding no matter what kind of hardware and storage you have.

Couchbase performance against disk is a work in progress. CouchDB started out 8X slower than SQLite as a backing store, apples to apples, but Couchbase is fixing that before they roll the product out. (After all, they wouldn’t want to slow the product down in the course of an upgrade.) Beyond that, when you do exploit the indexing capability of CouchDB, performance of course slows down. Work is underway to lower the performance hit; I imagine much improvement can indeed be made, given how few resources CouchDB has been able to devote to date to Bottleneck Whack-A-Mole.

Last week, Mike Stonebraker insulted MySQL and Facebook’s use of it, by implication advocating VoltDB instead. Kerfuffle ensued. To the extent Mike was saying that non-transparently sharded MySQL isn’t an ideal way to do things, he’s surely right. That still leaves a lot of options for massive short-request databases, however, including transparently sharded RDBMS, scale-out in-memory DBMS (whether or not VoltDB*), and various NoSQL options. If nothing else, Couchbase would seem superior to memcached/non-transparent MySQL if you were starting a project today.

*The big problem with VoltDB, last I checked, was its reliance on Java stored procedures to get work done.

Pleasantries continued in The Register, which got an amazing-sounding quote from Mike. If The Reg is to be believed — something I wouldn’t necessarily take for granted — Mike claimed that he (i.e. VoltDB) knows how to solve the distributed join performance problem. 

So, it’s Stonebraker against the web. And the difference of option is severe. In May, at a MongoDB developer conference in San Francisco, Mongo creator Dwight Merriman told his audience there was “no way” to do distributed joins in a way that really scales. “I’m not smart enough to do distributed joins that scale horizontally, widely, and are super fast. You have to choose something else. We have no choice but to not be relational,” he said

“You can do distributed transactions, but if you do them with no loss of generality and you do them across a thousand machines, it’s not going to be that fast.”

Stonebraker says precisely the opposite, and in typical fashion, he goes right for the jugular. “I reject what Merriman says out of hand,” he tells The Register. Merriman and his company, 10gen, declined to comment for this story. But Stonebaker says words don’t matter. As much as he likes to wield his opinions, he insists the debate will be decided elsewhere. “Let the bake-off begin,” he crows.

But when last I checked, VoltDB made nowhere near that claim. And well it shouldn’t have. In the fully general case, there’s no way to ensure super distributed join performance other than by throwing lots and lots of gear at the problem. But if you do that, many alternatives are fast. More specialized cases may be a different matter — but there are many fast alternatives for those too.

I imagine there will be use cases for which VoltDB sustains a lead as the truly fastest alternative, similarly-architected competitors perhaps excepted.* But what Mike supposedly said seems quite forward-leaning when compared to technical reality.

*The canonical VoltDB use case is e-commerce in virtual goods, the point of “virtual” being that physical inventory might necessitate costlier kinds of joins.

• 95% scalability up to 64 machines
• 90% scalability up to 88 machines
• 89% scalability up to 112 machines
• 84% scalability up to 128 machines

More precisely, those are counts of cluster “members,” but the recommended configuration is one member per operating system instance — i.e. one member per machine — for reasons of availability. In an 80% read/20% write workload, scalability is less — perhaps 90% scalability over 16 members.

Several elements are of IBM’s DB2 pureScale architecture are pretty straightforward:

• There are multiple pureScale members (machines), each with its own instance of DB2.
• There’s an RDMA (Remote Direct Memory Access) interconnect, perhaps InfiniBand. (The point of InfiniBand and other RDMA is that moving data doesn’t require interrupts, and hence doesn’t cost many CPU cycles.)
• The DB2 pureScale members share access to the database on a disk array.
• Each DB2 pureScale member has its own log, also on the disk array.

Something called GPFS (Global Parallel File System), which comes bundled with DB2, sits underneath all this. It’s all based on the mainframe technology IBM Parallel Sysplex.

The weirdest part (to me) of DB2 pureScale is something called the Global Cluster Facility, which runs on its own set of boxes.  (Edit: Actually, see Tim Vincent’s comment below.)These might have 20% or so of the cores of the member boxes, with perhaps a somewhat higher percentage of RAM (especially in the case of write-heavy workloads). Specifically:

• The DB2 pureScale Global Cluster Facility maintains a buffer pool (cache) shared by all the DB2 pureScale members.
• Even so, the DB2 pureScale members themselves are in charge of disk access.

So what’s going on here is not an Exadata-like split between database server and storage processing tiers. The Global Cluster Facility also handles lock management, presumably because locking issues only arise when a page gets fetched into the buffer.

The other surprise is that every client talks to every member, usually through a connection pool from an app server. Tim Vincent assures me that DB2 connections are so lightweight this isn’t a problem. Clients have load-balancing code on behalf of the members, and route transactions to whichever pureScale member is least busy.

DB2 pureScale is designed to be pretty robust against outages:

• In the case of planned maintenance, a pureScale member can be “quiesced.” I.e., it stops being given new work; it finishes up its existing work; then maintenance happens; then the member starts being given work again.
• In the case of an unplanned outage, the redo log naturally comes into play. The pureScale twist on this is that a second small instance of DB2 is around — or is started up? — just to handled the redos.

Also, IBM believes that the DB2 pureScale locking strategy gives availability and performance advantages vs. the Oracle RAC (Real Application Cluster) approach. The distinction IBM draws is that any member can take over the lock on a buffer page from any other member, just by attempting to change the page — and the attempt will succeed; only row-level locks can ever block work.  Thus, if a node fails, I/O can merrily proceed on other nodes, without waiting for any recovery effort. IBM’s target is <20 seconds for full row availability to be restored.

Obviously, it’s crucial that the Global Cluster Facility machines be fully mirrored, with no double failure — but so what? Modern computing systems have double-points-of-failure all over the place.

• Given Oracle’s market penetration and share, it makes sense that Oracle is focused on selling add-on products to its installed base. Oracle’s three top such go-to-market emphases at the moment are:
  • Database consolidation, especially on Exadata.
  • Data warehousing, presumably on Exadata.
  • Database security, especially encryption. This is not Exadata-specific, but does exploit Intel Westmere on-chip encryption, which Oracle says allows encryption with minimal overhead. This seems to be via something called Oracle Advanced Security.
• Deleted*

*Oracle asked me to delete a point on pricing they went out of their way to make, because they are in quiet period — even though nobody said it was confidential at the time, we weren’t under NDA, and it looks like public information to me anyway. Frankly, I’m not sure I was right to comply.

Oracle also told me quite a bit about Exadata onsite POCs (Proofs of Concept) and Exadata references, but I’ll save those subjects for future posts. The same goes for workload management.

Oracle’s version names and numbers can get confusing, but it turns out that:

• Oracle 11.203 11.2.0.3 will come out this fall. Oracle 11.204 11.2.0.4 will come out a little more than a year later. After that I imagine it will be time for Oracle 12.
• The current versions of Oracle Exadata are Exadata X2-2 and Exadata X2-8.
  • Oracle Exadata 2-2 is evolutionary from prior Exadata versions, and has 8 moderately big servers per rack. It can be sliced into half- or quarter-racks.
  • Oracle Exadata 2-8, in lieu of those 8 servers, has 2 bigger SMP (Symmetric MultiProcessing) systems, each with a terabyte of RAM. You can’t slice Exadata 2-8 below full-rack size, as you’d lose redundancy among the servers.

I didn’t really understand the discussion as to why certain workloads and/or workload consolidations go better on the SMP boxes of Exadata X2-8 than the blades of Exadata X2-2, but Oracle assures me that some do. I also suspect that some Oracle customers prefer large SMP boxes for no good reason other than familiarity.

As for recent-release adoption:

• Oracle estimates that 40-50% of customers have Oracle 11g running somewhere in their shops, mainly Oracle 11g Release 2.
• All major ISVs (Independent Software Vendors) are certified on Oracle 11g, typically Oracle 11g Release 2.
• But Exadata certification is something different from Oracle 11g certification; for example, SAP certification on Exadata is still underway, targeted for some time this year.

Exadata obviously enjoys huge performance gains over existing Oracle installations for certain analytic queries, and therefore for some whole analytic workloads. Oracle has happily trumpeted these. But it turns out that Exadata’s OLTP (OnLine Transaction Processing) performance gains are less dramatic. This makes all kinds of sense, given that Oracle’s analytic query performance was in pretty bad shape pre-Exadata, while OLTP has been just fine. The range Oracle used was 2-3X OLTP performance gains vs. existing Oracle installations on several-year-old hardware. Oracle says somewhere over 50% of Exadata physical I/O* goes against flash cache in uses cases such as running Oracle’s application suite.

*Note that physical I/O may be only a small fraction of logical; e.g., SAP long ago said that >99% of SAP transactions never hit disk.

Finally, we talked about a variety of options or other related products. Highlights included:

• One piece of the Oracle security story is a new product called Oracle Database Firewall, released in January, based on an acquisition of a small startup last year. Targeted primarily at internal hackers, Oracle Database Firewall sniffs your SQL traffic for a week or so, observes what kinds of SQL statements can be expected, builds a white list accordingly, and casts a jaundiced eye on any other kind of SQL statements that come through.
• Edit: I have no idea why I was told the following, in view of a subsequent email. Oracle Active Data Guard, first introduced in the Oracle 11g code line, is the preferred way to do active-active Oracle replication. That said:
  • Not a lot of customers use Oracle Active Data Guard yet …
  • … but a considerable fraction of Exadata users are at least interested in it.
  • Some number of Oracle customers have other kinds of active-active implementation. One option is via GoldenGate.
• Oracle Cloud File Management System is an Oracle 11g feature/option that lets you managed non-Oracle data. It is related to ASM (Automatic Storage Management), which seems to have been the most popular Oracle 10g feature, and which is essential to Exadata. Oracle Cloud File Management Systems seems to be popular for consolidation uses. But it is not technically well suited to, for example, play the role of HDFS in a MapReduce implementation.
• For DBAs who care, Exadata now supports Solaris on the database server tier as well as Linux. (That would be Solaris on Intel, of course; Exadata doesn’t use Sparc.) The storage tier still runs only on a kind of embedded Linux.
• Oracle 11g Express Edition (free crippleware) just went into beta test.
• And finally, Oracle SQL Developer 3.0 features, among other things, a GUI for Oracle Data Mining, and migration tools. Sybase migration is in there now, and was enhanced for SQL Developer 3.0. Teradata migration is slated for the next release.

MongoDB’s data model is based on BSON, which seems to be JSON-on-steroids. In particular:

• You just bang things into single BSON objects managed by MongoDB; there is nothing like a foreign key to relate objects. However …
• … there are fields, datatypes, and so on within MongoDB BSON objects. The fields are indexed.
• There’s a multi-value/nested-data-structure flavor to MongoDB; for example, a BSON object might store multiple addresses in an array.
• You can’t do joins in MongoDB. Instead, you are encouraged to put what might be related records in a relational database into a single MongoDB object. If that doesn’t suffice, then use client-side logic to do the equivalent of joins. If that doesn’t suffice either, you’re not looking at a good MongoDB use case.

MongoDB has integrated MapReduce. Natural uses include:

• The usual kinds of transformations one might do via MapReduce.
• Aggregations one might otherwise do in SQL (e.g. GROUP BY kinds of things are an obvious MapReduce fit).

Improved aggregation/MapReduce performance is a roadmap item.

However, Dwight said MongoDB has excellent performance in simple real-time reporting, for example updating a counter 10,000 times per second. When I asked him why, reasons included:

• A memory-mapped data model.
• Deferred writes — a write might take a couple of seconds to actually persist.
• Optimism — you don’t have to wait for an acknowledgement if you write something to the database.
• “Upsert in place” – update in place without checking whether you’re doing a write or insert.
• General lack of overhead.

Inspired in part by Schooner’s internal benchmarks, I’ve come to think that, apples-to-apples, even the simplest key-value store will have < 3X single-node performance advantage over well-implemented MySQL. (Read, write, or blended.) The 10gen guys don’t dispute that. However, they point out that a single MongoDB request can process the equivalent of many relational rows, opening the possibility of much greater performance gains than that. In particular, there seem to be some Drupal implementations enjoying huge MongoDB-based speed-ups.

On a heavy-duty server (8-12 cores, 16-64 gigs of RAM), MongoDB can apparently do 20-30,000 writes or 100,000 reads per second. Improved concurrency and mixed read/write performance is coming, although obviously 10gen would think that MongoDB does pretty well in those areas already. The largest known MongoDB system does about 1 million reads/second. I would imagine that those figures require the database to fit into RAM, given:

• MongoDB’s memory-mapped architecture.
• This post mortem of the MongoDB Foursquare outage.

I’ve gotten conflicting signals as to whether there are any multi-hundred-node MongoDB deployments. But there are “lots” of 20-40 node ones, as well as lots under 10 nodes. Note that 1 million reads/second naively sounds as if it could be achieved on 10 MongoDB nodes — but I’d guess that’s not really the configuration.

MongoDB’s scale-out story starts with transparent sharding, although I think transparent sharding was a new feature in MongoDB 1.6 last summer. So far as I understand, what MongoDB sacrifices in the CAP Theorem is the P — partitions happen, and when they do you might not be able to write to the shard you want to. A MongoDB shard can have multiple nodes, in a master-slave set-up, and there’s some flexibility as to how intra-node consistency is handled.

MongoDB functionality futures include full-text search, and extensions to the general query language.