Reporter: [Care to comment]?
CAM: SQL Server is an adequate product if you don’t mind being locked into the Microsoft stack. For example, the ColumnStore feature is very partial, given that it can’t be updated; but Oracle doesn’t have columnar storage at all.

Reporter: Is the lock-in overall worse than IBM DB2, Oracle?
CAM: Microsoft locks you into an operating system, so yes.

Reporter: Is this release something larger Oracle or IBM shops could consider as a lower-cost alternative a co-habitation scenario, in the event they’re mulling whether to buy more Oracle or IBM licenses?
CAM: If they have a strong Microsoft-stack investment already, sure. Otherwise, why?

Reporter: [How about] just cost?
CAM: DB2 works just as well to keep Oracle honest as SQL Server does, and without a major operating system commitment. For analytic databases you want an analytic DBMS or appliance anyway.

Best is to have one major vendor of OTLP/general-purpose DBMS, a web DBMS, a DBMS for disposable projects (that may be the same as one of the first two), plus however many different analytic data stores you need to get the job done.

By “web DBMS” I mean MySQL, NewSQL, or NoSQL. Actually, you might need more than one product in that area.

• PostgreSQL is good technology.
• MySQL is narrowing the gap, but PostgreSQL is still ahead of MySQL in some ways.  (Database extensibility if nothing else.)
• PostgreSQL has a lot of users. (Many of them in academia and/or Russia.)
• Neither EnterpriseDB (which now calls itself “The enterprise PostgreSQL company”) nor the PostgreSQL community leadership have covered themselves with stewardship glory.
• A significant number of interesting DBMS products can be regarded as PostgreSQL forks (e.g. Greenplum, Aster Data nCluster, Netezza if you squint, and Vertica if you stand on your head*).
• PostgreSQL advancement is not dead. For example, Hadapt beta users are running actual PostgreSQL on many nodes each.
• There’s no assurance that Oracle will be a benevolent MySQL steward forever. (Specifically, Oracle’s “Play nicely with others” antitrust commitments expire in 2014.)

So I think it would be cool if one or the other big company put significant wood behind the PostgreSQL arrow.

*While Vertica was originally released using little or no PostgreSQL code — reports varied — it featured high degrees of PostgreSQL compatibility.

My favorite relational OLTP scale-out choice these days is the SchoonerSQL/dbShards partnership. Schooner Information Technology (SchoonerSQL) and Code Futures (dbShards) are young, small companies, but I’m not too concerned about that, because the APIs they want you to write to are just MySQL’s. The main scenarios in which I can see them failing are ones in which they are competitively leapfrogged, either by other small competitors – e.g. ScaleBase, Akiban, TokuDB, or ScaleDB — or by Oracle/MySQL itself. While that could suck for my clients Schooner and Code Futures, it would still provide users relying on MySQL scale-out with one or more good product alternatives.

Relying on non-MySQL NewSQL startups, by way of contrast, would leave me somewhat more concerned. (However, if their code is open sourced. you have at least some vendor-failure protection.) And big-vendor scale-out offerings, such as Oracle RAC or DB2 pureScale, may be more complex to deploy and administer than the MySQL and NewSQL alternatives.

The short form of the SchoonerSQL (as Schooner’s product is now called) story goes roughly like this:

• SchoonerSQL replicates data — synchronously if the replication target is local, asynchronously if it is remote.
• Local synchronous replication provides high availability; remote asynchronous replication provides disaster recovery.
• SchoonerSQL’s local synchronous replication also provides read scale-out.
• Schooner has a partnership with Code Futures/dbShards to provide write scale-out via transparent sharding.
• SchoonerSQL has some secret sauce in replication performance. This has the effect of significantly increasing write performance (assuming you were going to replicate anyway), because otherwise you might have to slow down the master server’s write performance so that the slaves can keep up with it.
• Schooner believes it still has some single-server performance advantages as well.

Just to be clear here: Schooner is my client. Code Futures is my client. I introduced them and suggested their partnership. I then introduced them both to a user client, who was sufficiently impressed to seriously evaluate both of them. Even so, some of the Schooner replication story only became clear to me when I visited last Friday.

To flesh that out a bit more:

• Schooner has been making performance tweaks to the MySQL/InnoDB stack for several years. Some of them are still relevant, and can offer 50-100% performance improvements, especially but not only if you use solid-state storage.
• Schooner has found a way to scale up the slave side of master-slave replication, both in the synchronous and asynchronous cases.
  • The core idea of SchoonerSQL parallel (i.e. scale-up) replication is straightforward. The replication log streams in. A chunk is sent off to a CPU core. The next chunk is examined for conflicts with the first chunk. If there are none, it is sent off to a different core to be processed
  • Thus, you can have both local replication/high availability and remote replication/disaster recovery without slowing down writes on the master the way you may have to with other alternatives.
  • Schooner believes this feature alone can make SchoonerSQL 3X faster than MySQL alternatives, at least for writes.

At least in casual conversation, Schooner synthesizes its 1.5-2X single-server figure and up-to-3X clustering figure into a single claim of frequent 2-5X speedup over generic MySQL/InnoDB. But the usual caveats about vendor-supplied performance numbers of course apply.

Finally, some housekeeping:

• At Oracle’s polite request,* Schooner changed its product name to not mention MySQL; hence the moniker SchoonerSQL.
• SchoonerSQL is being launched Monday.
• Schooner has determined that if its version numbers are different from MySQL’s, confusion ensues. So the first ever version under the name SchoonerSQL is SchoonerSQL 5.1, a factoid that Schooner is wisely omitting from the official product launch press release.
• The synchronous part of SchoonerSQL’s replication story dates back to last January’s product release.
• Most of the asynchronous part of SchoonerSQL story is new for Monday.
• dbShards/SchoonerSQL partnership engineering is still underway.

*I mean that non-facetiously. Schooner’s MySQL OEM contract was such that Oracle didn’t have a legal hammer to force the change. Edit: Whoops! There turn out to have been inaccuracies in the original version of this footnote, which I now regret writing. The contract isn’t exactly OEM, and there actually were some trademark-based legal hammers.

NoSQL is most easily defined by what it excludes: SQL, joins, strong analytic alternatives to those, and some forms of database integrity. If you leave all four out, and you have a strong scale-out story, you’re in the NoSQL mainstream.

• Thus, I’d say Cassandra, HBase, Mongo DB, and Couchbase are prime examples, in no particular order. Riak as well.
• I might have phrased that better if I’d used a different word than simply “strong” — but hey, there was a 140-character limit, and he was on deadline.

Using NoSQL can make sense when at least one of two things is paramount: low-cost scale-out or dynamic schemas.

• There are some seriously sensible use cases for dynamic schemas.
• “Low-cost” generally boils down to:
  • Performance.
  • Open source free-like-beer.
  • Not a lot of database administration.

I’ve generally given object-oriented DBMS vendors and also MarkLogic hard times whenever they consider saying they’re “NoSQL”. Reasons include:

• Closed source.
• Database administration overhead (even if you get good stuff for incurring that overhead, like MarkLogic’s comprehensive indexing).

Also, NoSQL started out being ACID-unfriendly.

What you give up are the query flexibility and the easily automatic data integrity of SQL-based systems. I should have added something about a mature ecosystem.

In the most recent live example, I influenced a client away from Cassandra and toward scale-out MySQL (dbShards and/or Schooner flavors, most likely). Part of the reason was the ability to do joins, which are useful in their application. Another part is that their development practices obviated any significant benefit from dynamic schemas. But perhaps the most important — or at least resonant — reason of all was that they really, really cared about .NET support.

There really are three major points to NoSQL.

• Dynamic schemas. This is the only one of the three that truly depends on NoSQL.
• Scale-out short-request processing. If you want to scale out efficiently at high request volumes, you’re best off not using all the flexibility SQL/relational DBMS offer. (In particular, you don’t want to do cross-node joins). Not coincidentally, a number of the best scale-out offerings were built to be NoSQL.
• Open source. Doing a relational DBMS is a big project. It seems easier to build NoSQL ones.

Oracle can address the latter two points as aggressively as it wishes via MySQL. It so happens I would generally recommend MySQL enhanced by dbShards, Schooner, and/or dbShards/Schooner, rather than Oracle-only MySQL … but that’s a detail. In some form or other, Oracle’s MySQL is a huge player in the scale-out, open source, short-request database management market.

So that leaves us with dynamic schemas. Oracle has at least four different sets of technology in that area:

• As Workday noticed years ago, MySQL can be used as a functional, basic key-value store.
• Oracle also has XML-based Berkeley DB/SleepyCat kicking around.*
• The XML extensions to Oracle’s core DBMS could be alleged to have a dynamic schema/NoSQL flavor. (Blech.)
• A dynamic schema argument could also be made for object-oriented DBMS technology. While Oracle doesn’t to my knowledge exactly sell that, it does have the Tangosol Coherence line of technology, with a potentially similar programming model.

If Oracle is now refreshing and rebranding one or more of these as “NoSQL”, there’s no reason to view that as a big deal at all.

*That’s Mike Olson’s former company, if you’re keeping score at home.

I’m not hugely optimistic about the Oracle Database Appliance. Rather, my thoughts — lightly edited from a chat with a reporter — include:

• This doesn’t solve Oracle’s SMB problems, which include:
  • Oracle software is too difficult and costly to administer. The appliance will make a modest dent in that one, but it’s not any kind of game-changer, because the issues relate to the antique design of the Oracle DBMS. (I.e., I think ongoing database administration is a bigger deal than, say, one-time system set-up.)
  • SMBs use third-party applications whenever they can, with an increasing preference for SaaS.  Application and SaaS vendors prefer non-Oracle alternatives when they are feasible.
• Thus, Oracle is not well positioned to thrive in the SMB market … except maybe through its MySQL subsidiary, but that has a long way to go too.
• Clayton Christensen’s The Innovator’s Solution teaches us that Oracle should focus on selling a thick stack of technology to its highest-end customers, and that’s exactly what Oracle does focus on.

• 100s of gigabytes of data at first, growing to >1 terabyte over time.
• High peak loads.
• Public cloud portability (but they have private data centers they can use today).
• Simple database design — not a lot of tables, not a lot of columns, not a lot of joins, and everything can be distributed on the same customer_ID key.
• Stream the data to a data warehouse, that will grow to a few terabytes. (Keeping only one year of OLTP data online actually makes sense in this application, but of course everything should go into the DW.)

So I’m leaning to saying:  

• They should go with a scalable, MySQL-based solution.
  • Lots of third-party software works with MySQL, in case that’s helpful.
  • Yes, any one vendor is small and not yet firmly established, but there are numerous vendors around with interesting MySQL scaling stories.
  • In a vendor emergency, just going with Oracle’s MySQL stuff would probably work …
  • … especially because there are these lovely things in the world called solid-state drives.
  • There’s also good escapability if one wants to move away from MySQL, because everybody knows how to handle MySQL data.
• The first product to look at is dbShards, because it meets all the topology needs:
  • Local scale-out (transparent sharding).
  • Local high availability.
  • Remote disaster recovery (details of that are underway).
• The first analytic DBMS to look at is Infobright.
  • Yes, I know Infobright is focused more on machine-generated data these days, but this client’s analytic needs are so straightforward Infobright should pass with flying colors.
  • The MySQL-to-MySQL aspect should make ETL dead simple.
  • Again, there’s escapability.

Mainly, this is all fine. But I’m getting pushback on the solid-state aspect, for fear that it will compromise public cloud portability.

Am I missing something here? As far as I’m concerned, if you’re planning an OLTP system with a many-year lifespan today, of course you should assume solid-state storage. Maybe you scale out just as far as you would with disk, striping indexes or entire databases across the RAM of multiple servers. It that case, having solid-state backing reduces the risk of bottlenecks. Maybe you don’t scale out as far as you would with disk. In that case, solid-state backing saves you money.

As for public-cloud support for solid-state storage, that’s coming fast, right? (Actually, I have data points in support of that theory, but they’re a bit tenuous.) A large fraction of web businesses with private data centers seem to be using solid-state storage — from Facebook on down — or so the NoSQL/NewSQL/short-request DBMS guys tell me. Surely a number of public cloud vendors are close behind.

memcached is a way to cache data in RAM across a cluster of servers and have it all look logically like a single memory pool, extremely popular among large internet companies. The Membase product — which is what Couchbase has been selling this year — adds persistence to memcached, an obvious improvement on requiring application developers to write both to memcached and to non-transparently-sharded MySQL. The main technical points in adding persistence seem to have been:

• A persistent backing store (duh), namely SQLite.
• A change to the hashing algorithm, to avoid losing data when the cluster configuration is changed.

Couchbase is essentially Membase improved by integrating CouchDB into it, with the main changes being:

• Changing the backing store to CouchDB (duh). This will be in the first Couchbase release.
• Adding cross data center replication on CouchDB’s consistency model. This will not, I believe, be in the first Couchbase release.
• Offering CouchDB’s programming and query interfaces as an option. So far as I can tell, this will be implemented straightforwardly in the first Couchbase release, with elegance planned for later down the road.

Let’s drill down a bit into Membase/Couchbase clustering and consistency.

• When data is written to RAM in memcached, it immediately gets copied to another server. The same is of course true in Membase/Couchbase. The terminology on all this is confusing, but I think:
  • The portion of data that is stored as a primary copy on any given server is called a “shard”.
  • That would seem to make sense, as that data could correspond to what goes — non-transparently — into an instance of MySQL in a classical memcached/MySQL set-up.
• Updates are of course also banged to disk ASAP — but at times of heavy load, that can take a while. A few seconds to a couple of minutes is normal operation; if it takes an hour, you really should buy more hardware. (Or solid-state storage.)
• Similarly, the replication of data to a second machine’s RAM may not happen at times of heavy load — and that’s another sign you don’t have enough machines.
• Each Membase/Couchbase “shard” has lots of logical sub-shards.* (1024 for now, at least as default, although Dustin finds that number excessive and is looking to lower it.)  So if you add a node, some of the sub-shards get sent over to the new node. Unlike the case for straight memcached, no data is lost from cache (and of course not also from the persistent store). Blocking of operations from such a move only happens in narrow time windows, and then only in edge cases.

*Edit: They’re called vbuckets.

So if we consider Membase technology alone, Couchbase is CA in the CAP Theorem.  CouchDB, however, is gloriously AP in the CAP Theorem, in that it was written to assume an occasionally connected topology.* Based on that, Couchbase will allow AP operation between data centers (i.e. “stay synchronized if you can, to within the limitations of physics and so on, but don’t beat yourself up on the rare occasions that you can’t.”) I don’t know that that capability will quite be in the first release of Couchbase, but it’s coming soon.

*CouchDB also has other features friendly to occasionally-connected use cases, such as a lot of flexibility as to which parts of the database are or aren’t synced when you do reconnect. These are at the heart of the Couchbase Mobile offering.

memcached and Membase have a very simple key-value interface. CouchDB adds secondary indexes and so on. I think in the first release of Couchbase this is pretty much like having two different APIs for the same product; more elegant integration is planned down the road, and more language support as well.

The highest-performing way to use Couchbase will probably always be to just pretend it is Membase, which is to say memcached+. Dustin told me of Membase users who demanded 10-40 millisecond response times, and that not even for single queries but rather for sequences of several queries in succession. He further told me of customers asking for 1-200 microsecond response, and insisting on no worse than 1 millisecond. Frankly, the first requirement could be met by lots of technologies I can think of, at least if  you don’t rely on disk; the second is thoroughly impossible if you rely on disk, and pretty demanding no matter what kind of hardware and storage you have.

Couchbase performance against disk is a work in progress. CouchDB started out 8X slower than SQLite as a backing store, apples to apples, but Couchbase is fixing that before they roll the product out. (After all, they wouldn’t want to slow the product down in the course of an upgrade.) Beyond that, when you do exploit the indexing capability of CouchDB, performance of course slows down. Work is underway to lower the performance hit; I imagine much improvement can indeed be made, given how few resources CouchDB has been able to devote to date to Bottleneck Whack-A-Mole.

Context for any comments on customer traction includes:

• Membase went into limited production release in October, and full release in January. Similar things are true of CouchDB.
• Hence, most sales of Couchbase’s products have been made over the past 6 months.
• Couchbase (the merged product) is at this point only in a pre-production developer’s release.
• Couchbase has both a direct sales force and a classic open-source “funnel”-based online selling model. Naturally, Couchbase’s understanding of what its customers are doing is more solid with respect to the direct sales base.
• Most of Couchbase’s revenue to date seems to have come from a limited number of big-ticket “lighthouse” accounts (as opposed to, say, the larger number of smaller deals that come in through the online funnel).

That said,

• Most Membase purchases are for new applications, as opposed to memcached migrations. However, customers are the kinds of companies that probably also are using memcached elsewhere.
• Most other Membase purchases are replacements for the Membase/MySQL combination. Bob says those are easy sales with short sales cycles.
• Pure memcached support is a small but non-zero business for Couchbase, and a fine source of upsell opportunities.
• In the pipeline but not so much yet in the customer base are SaaS vendors and the like who use and may want to replace traditional DBMS such as Oracle. Other than among those, Couchbase doesn’t compete much yet with Oracle et al.
• Pure CouchDB isn’t all that much of a business, at least relative to community size, as CouchDB is a single-server product commonly used by people who are content not to pay for support.

Membase sales are concentrated in five kinds of internet-centric companies, which in declining order are:

• Social gaming
• Ad platforms
• Online retail
• Online business, including B2B  SaaS
• Social networking

Bob said that Couchbase often sees MongoDB competitively, but never Riak, HBase, or Redis. I got the impression Couchbase sees at least a little Cassandra. That would, of course, all pertain only to direct sales, rather than download/community kinds of usage.

Couchbase is also excited about the potential for the CouchDB-based Couchbase Mobile occasionally-connected offering. The hottest use cases, interestingly, seem to be non-consumer; Bob rattled off military, farming, and health care, and surely could have named more besides. However, the Couchbase Mobile sales effort still seems to be in early days, as is evidenced by the fact that Couchbase has not yet competitively encountered Sybase SQL Anywhere.

With all that said, I’ll go now to a separate post for a Couchbase technical update.