• Facebook et al. are in effect Software as a Service (SaaS) vendors, not enterprise technology users. In particular:
  • They have the technical chops to rewrite their code as  needed.
  • Unlike packaged software vendors, they’re not answerable to anybody for keeping legacy code alive after a rewrite. That makes migration a lot easier.
  • If they want to write different parts of their system on different technical underpinnings, nobody can stop them. For example …
  • …  Facebook innovated Cassandra, and is now heavily committed to HBase.
• It makes little sense to talk of Facebook’s use of “MySQL.” Better to talk of Facebook’s use of “MySQL +  memcached  + non-transparent sharding.” That said:
  • It’s hard to see why somebody today would use MySQL +  memcached  + non-transparent sharding for a new project. At least one of Couchbase or transparently-sharded MySQL is very likely a superior alternative. Other alternatives might be better yet.
  • As noted above in the example of Facebook, the many major web businesses that are using MySQL +  memcached  + non-transparent sharding for existing projects can be presumed able to migrate away from that stack as the need arises.

Continuing with that discussion of DBMS alternatives:

• If you just want to write to the memcached API anyway, why not go with Couchbase?
• If you want to go relational, why not go with MySQL? There are many alternatives for scaling or accelerating MySQL — dbShards, Schooner, Akiban, Tokutek, ScaleBase, ScaleDB, Clustrix, and Xeround come to mind quickly, so there’s a great chance that one or more will fit your use case. (And if you don’t get the choice of MySQL flavor right the first time, porting to another one shouldn’t be all THAT awful.)
• If you really, really want to go in-memory, and don’t mind writing Java stored procedures, and don’t need to do the kinds of joins it isn’t good at, but do need to do the kinds of joins it is, VoltDB could indeed be a good alternative.

And while we’re at it — going schema-free often makes a whole lot of sense. I need to write much more about the point, but for now let’s just say that I look favorably on the Big Four schema-free/NoSQL options of MongoDB, Couchbase, HBase, and Cassandra.

… start-ups that are building technologies to enable MySQL and other SQL databases to get over some of the problems they have in scaling past a certain size. … I’d like to get a sense as to whether or not the problems are as severe and wide spread as these companies are telling me? If so, why wouldn’t a customer just move to a new database?

While that sounds as if he was asking about scale-out relational DBMS in general, MySQL or otherwise, short-request or analytic, it turned out that he was asking just about short-request scale-out MySQL. My thoughts and comments on that narrower subject include(d) but are not limited to: 

• The biggest web companies had to go to non-transparently sharded MySQL years ago. The NoSQL movement is, in no small part, an attempt to improve upon that. Ditto for scale-out short-request MySQL.
• Some overlapping categories of companies or projects who need scale-out short-request database processing are:
  • The aforementioned big companies who have other applications they haven’t hand-sharded yet.
  • Other web companies whose applications are getting that big.
  • Conventional enterprises whose web efforts happen to be very big.
  • Sensor networks and other massive sources of machine-generated data.
  • Certain specialized areas (e.g., financial trading).

• Relatively few of these applications are totally impossible to do in Oracle. But the Oracle approach might be very expensive.
• In particular, there’s a break point when companies — often SaaS vendors — outgrow Oracle Standard Edition.
• Yes, the alternatives usually are one of MySQL or Oracle.
• InnoDB isn’t an alternative to these newer technologies; it’s just a piece of the puzzle and indeed of default MySQL now. Several of them — e.g. dbShards — are meant to be used in conjunction with InnoDB.
• Merging his list and mine, the high-performance/scale-out MySQL alternatives look like dbShards, Schooner, ScaleBase, ScaleDB, Tokutek, Akiban, Xeround, and Clustrix. The first two are to my knowledge more proven than the rest.
• Proprietary hardware and the associated hardware/appliance pricing aren’t very appealing for these applications. That speaks against Oracle Exadata and Clustrix, and is the reason Schooner switched to a software-only strategy despite some initial appliance sales.
• However, hardware band-aids such as solid-state drives or even RAM-based solid-state storage could make more sense:
  • If, for performance, you’ve scaling out your database so that it fits in RAM on each box, you don’t really have a disk-based architecture anyway, now do you?
  • Even if you’re not doing that yet — if your problem is throughput rather than storage capacity, silicon-based storage could be a big help.
  • In principle, devices of that kind can be moved from one application to another, after the first one is rearchitected not to need them. (In practice, however, I don’t know of anybody who is doing that. I also don’t believe that Kaminario et al. are marketing that kind of idea, more’s the pity.)
• My notes on all this from April, 2010 are already badly outdated, but may be interesting anyway.

• Dwight Merriman of 10gen (MongoDB)
• Damien Katz of Couchio (CouchDB)
• Matt Pfeil of Riptano (Cassandra)
• Todd Lipcon of Cloudera (HBase committer)
• Tony Falco of Basho (Riak)
• John Busch of Schooner
• Ori Herrnstadt of Akiban

By no means do I have time to do these conversations justice, in terms of giving them the write-ups and/or immediate follow-up that they deserve. Indeed, I’ll leave for vacation Saturday morning with my 2000-word NoSQL article still unwritten. So I’ll dump as many observations as I can into one or a few posts now, and play catch-up later as circumstances allow.

In no particular order:

• A number of NoSQL offerings have had more uptake to date than most of the scale-out SQL offerings have.
• “Document-oriented” NoSQL projects CouchDB and MongoDB have probably had the most users get into production, but perhaps for pretty small systems.
• Cassandra and Hbase — the column-group-architecture guys — have probably had the most bang-in-lots-of-writes HVSP production uptake.*
• I didn’t talk customer count with Schooner, but the decently-stocked Schooner customer page suggests Schooner may be something of an exception to these generalities.
• A lot of these companies are in the low-to-mid-teens of employees.
• The SQL-oriented companies, despite having fewer or no customers, often seem to have more money. (One reason I get the impression SQL guys have more money is, frankly, that more of them are talking about engaging my services.)
  • Schooner cites $20 million in VC.
  • Clustrix cites a figure close to that.
  • Basho cites $10 million, plus a new round of $1.5 or $2 or $2.5 million. The new round is at a lowered valuation.
  • That same site says Tokutek finally was able to raise some VC. Congrats!
• It’s only a two-company trend, but I was pleased to hear that both 10gen/MongoDB and Akiban were seeing Drupal as a major use case or potential use case. No word on rescuing WordPress from its MySQL implementation, alas, but it seems that a Drupal site typically has 40-200+ tables, while a WordPress one has 10ish.
• Another trend I think I’m seeing is serious object-oriented apps banging things straight into a simple back end. Workday is a huge example of that. Akiban hopes to do something similar with Hibernate.
• Stability and maturity are still issues for many of these products. E.g., HBase isn’t even in Release 1.0 yet. Ditto Cassandra, and surely many of the others. Unsurprisingly, making Cassandra stable is still a challenge.

*As is common for terms I suggest, the “HVSP” name is not getting any traction. What do you think of Marton Trencseni’s suggestion of OLRP, for OnLine Request Processing?

One thing that makes following this area interesting is that so many projects are open source, leading there to be a lot of information in the wild. I hardly have time to read the mailing list for each project; but the people I talk with do, and often they may sorta kinda remember something somebody else posted one or several months back. As just one example, the mailing lists are said to confirm:

• Contrary to rumor, Facebook hasn’t moved in-box search off of Cassandra.
• Apparently, however, it’s true that Cassandra inventor Facebook has stopped working on Cassandra, and Facebook’s core Cassandra developers have shifted over to HBase.

Also, figuring out usage of open source software can be … interesting.

• People who use open source software don’t have to reveal themselves, as there’s no purchase transaction to kick things off.
• On the other hand, if they’re serious enough in their use, they often do.
  • There are two main ways to get tech support for open source software — the community or a company that sells support — and both ways let the main support-selling company know that one is a user.
  • Some folks even add themselves to open lists of users, for example these rather long lists for HBase and CouchDB.
  • Or they show up at conferences. For example, two tweets from Riptano founder Jonathan Ellis suggest at least 30 production Cassandra users were represented at a recent event. That’s more detail than his colleague Matt Pfeil wanted to give me when talked.

OK. This post has gotten pretty long, even without me saying anything resembling an overview of any of the seven companies I listed up top, or of their products’ adoption. So I’ll just publish this now.

It also is understood that, realistically, I can’t be expected to know and mention the very latest news for all the many products in the categories. Even so, I think this would be fine time to check just where NoSQL and HVSP adoption stand. Here is most of what I know, or links to same; it would be great if you guys would contribute additional data in the comment thread.

In the NoSQL area: 

• Back in April, the VoltDB guys told me they thought Cassandra and HBase were the two NoSQL systems with the most momentum.
• I know distressingly little about HBase adoption, but a source who may or may not wish to remain anonymous was kind enough to alert me that Twitter and StumbleUpon each have ~30 node deployments, for analytics and analytics/HVSP respectively.
• I wrote in detail on Cassandra adoption last month. News since then includes:
  • Facebook is rumored to have dropped Cassandra completely.
  • Twitter clarified that it may not be quite as lovestruck by Cassandra as before, but they’re still very close friends.
  • It’s not obvious that the Cassandra Summit unveiled a lot of new adoption stories.
• Northscale’s Membase is still in its early days.  Zynga is bought in, however, as is something called NHN Korea. (Edit: I subsequently saw NHN Korea on a prominent SEO expert’s list of the top half dozen or so search engines in the world. Who knew?)
• Basho has listed a few Riak customers. If memory serves (I haven’t spoken with Basho for a while, and some of my notes are misplaced due to some computer sloppiness), Basho has a few dozen customers in total.
• Mozilla has a 4 machine, 64 core Riak cluster in production.
• Hypertable has a few users/project sponsors, Baidu being the biggest name among them.
• I don’t really know how the MongoDB/10gen guys are doing. I think this is at least as much my fault as theirs. Anyhow, they seem to have links to a couple of folks who have written about MongoDB usage.
• NimbusDB is still in stealth mode. I’d be surprised if they had users for a while yet, since in January they didn’t yet sound as if development was very far underway. (Actually, I forget whether NimbusDB is supposed to be SQL-based or not.)

Among the SQL or SQL-friendly guys:

• Clustrix says it has a few production users, some big-name, but is not disclosing them yet.
• dbShards has around 6 customers, including Facebook. (Facebook may outpace even Twitter and Zynga in using the most products mentioned in this post.)
• As of May, VoltDB had one paying customer, plus 150 beta customers who weren’t in production yet.
• Akiban says they’ll get me up to speed on Thursday.
• ScaleDB seems to be pedaling along in perennial beta. Whether ScaleDB has any actual beta users is less clear. On the plus side, checking that out uncovered a pretty funny April Fool blog post.
• Groovy Corporation seems to have disappeared, or morphed into something called uCirrus, or something like that.

Nonetheless, OLTP/general-purpose data management startup activity has recently picked up, targeting what I see as some very real opportunities and needs. So as a jumping-off point for further writing, I thought it might be interesting to collect a few observations about the market in one place.  These include:

• Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.
• By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value.
• Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.
• It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.
• The era of silicon-centric relational DBMS is coming.
• The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds.
• Users’ instance on “free” could be a major problem for OLTP DBMS innovation.

I shall explain.

Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.

• OLTP applications are more complex than analytic ones, and hence more tightly wired into particular brands of DBMS. For example, third-party packaged OLTP applications are typically portable among only a few brands of DBMS. But third-party business intelligence tools, and the BI “applications” built in them, are more easily and widely portable.
• Specific technical observations such as “OLTP apps tend to use stored procedures, which are DBMS-specific” or “OLTP apps tend to have lots and lots of tables” serve to underscore the first point.
• An enterprise’s highest-value data is commonly the financial stuff handled by its core OLTP systems, so those are the last things they want to mess around with just to get some cost savings. Security, high availability, and so on are major considerations that can outweigh cost.

By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value. Indeed, “OLTP” is often a misnomer, which is why I tend to go with “general-purpose” or some similarly wishy-washy phrase instead.

• In theory, this is a ripe area for what I’ve called mid-range DBMS.
• The big brand vendors try hard to keep as many of those databases for themselves as they can. Enterprise-wide license pricing helps. Going forward, so will virtualization/consolidation strategies, such as Oracle’s Exadata-centric approach.
• A variety of mid-range DBMS alternatives beyond the big brands have technical merit, at least in some cases and configurations – MySQL, PostgreSQL, Intersystems Cache’, and so on.
• The only such mid-range DBMS alternative with much large enterprise business momentum, however, appears to be MySQL.

“General-purpose” might be a better term than “OLTP” anyway.

• I don’t have a link, but it’s widely agreed that over half of the processing on an “OLTP” enterprise app is commonly reporting and so on.
• “Operational BI” is progressing by fits and starts, but it is progressing.
• Anything customer-facing — web-based, call center, or otherwise — is likely to include a heavy dose of “real-time” analytic optimization.

Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.

• VoltDB is the main exception that jumps to mind.
• This isn’t true in the analytic DBMS area, where Netezza, Greenplum, Aster, Vertica and others started from PostgreSQL’s code, APIs, or both.

It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.

• MySQL is a limited product without all the third-party storage engines that are being developed.
• Oracle’s promise of MySQL good behavior has an expiration date.
• None of the MySQL front-end alternatives are remotely mature yet.

The era of silicon-centric relational DBMS is coming.

• I think “silicon” means “solid-state memory” as much as or more than it means “RAM,” but that’s not yet certain.
• What is pretty certain is that, thanks to Moore’s Law, some kind of silicon will increasingly replace disk.
• Oracle’s increasingly Flash-centric story is a challenge to everybody.
• RAM-centric VoltDB will launch fairly soon. (By the way, while VoltDB still has a lot in common with H-Store, they’re not exactly the same thing. And H-Store research is progressing too.)
• RethinkDB is being developed, focused directly on solid-state memory. Based on the sparse information available online, RethinkDB sounds somewhat like a dumbed-down H-Store.
• New disk-based vendors may never optimize their use of disk, instead targeting a solid-state future. (E.g., I think Akiban should and quite well might follow this path.)

The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds. We hear that from the NoSQL guys all the time. But I also just heard it from Akiban.

Users’ instance on “free” could be a major problem for OLTP DBMS innovation. Vendors of new OLTP data management technologies often feel obligated to open source their products, notwithstanding the historical lack of revenue in the open source OLTP DBMS market. As just one of many examples, Nova Spivack wrote:

I have recently seen some new graph data storage products that may provide the levels of scale and performance needed, but pricing has not been determined yet. In short, storage and retrieval of semantic graph datasets is a big unsolved challenge that is holding back the entire industry. We need federated database systems that can handle hundreds of billions to trillions of triples under high load conditions, in the cloud, on commodity hardware and open source software. Only then will it be affordable to make semantic applications and services at Web-scale.

I hear similar things from other startups, who evidently believe they need and/or are entitled to enjoy sophisticated, high-performance, zero-cost, specialized database management technology.

• Akiban is basically in the business of making OLTP (OnLine Transaction Processing) DBMS.
• That said, Akiban does not necessarily aspire to offer the DBMS that has the best update efficiency or throughput. In particular, the Akiban DBMS stores every datum twice, even before replication (and indexes) are taken into account.
• Akiban wants to store everything as third normal form relational databases. I didn’t ask whether 3NF is a hard requirement, a Really Good Idea if you want Akiban to run fast (that’s the one I’d guess), or merely a general design assumption.
• Akiban characterizes its core differentiators/value proposition as being:
  • Scale-out
  • No need to pay the traditional cost of joins
• Thus, Akiban is telling something like a NoSQL story.
• However, Akiban offers SQL.
• Specifically, Akiban offers SQL through a MySQL front end. However, the choice of front-end could change (Drizzle?), and non-relational front-ends (object?)* could eventually also be offered.
• Akiban’s first target market is SaaS providers, specifically ones that have true multitenancy issues. More generally, Akiban is pursuing cloud/private cloud applications with lots of tables. (Ori talks of a few thousand tables as being a small number.) At least at first, Akiban is conceding the market for huge-volume, scale-out, no-expensive-join web databases to the NoSQL contenders.
• Akiban has been in prototyping/development of some kind for several years. However, Akiban got its first angel funding early last year and its first venture funding late in 2009, so development only ramped up recently.
• Ori tells a version of the rather common “Everything I need to know in life I learned in the Israeli Army, and now I’m commercializing it” story. However, I didn’t get the sense that Akiban is necessarily a direct extension of a specific Israeli military project.

* A lot of Boston-area DBMS developers have significant non-relational experience. E.g., Jack Orenstein was an Object Design founder, and Peter Beaman used to work for Intersystems, both object-oriented DBMS vendors.

Akiban technical highlights include:

• Somewhat confusingly, Akiban databases are divided into “groups” of tables. The point of Akiban groups is:
  • Many-to-many relationships exist only within Akiban groups, not among tables in different groups.
  • Tables within Akiban groups are kind of pre-joined; more precisely, data is organized physically in a way that anticipates joins.
  • Thus, most Akiban joins can be executed without the cost of traditional join algorithms.
• One copy of the data Akiban stores is, in effect, clustered by object. E.g., a customer and her orders are stored together, or a patient and the records of her doctor visits. That’s how Akiban anticipates most joins.
• The other copy of Akiban data is stored in columns (I’m not sure if this part is strictly columnar or more hybrid row/column), which are ordered consistently. In particular, they’re in an order dictated by the organization of the other copy of the data, whatever that means. Akiban’s goal is for this copy of the data to support reporting, operational BI, etc.
• Akiban relies heavily on its optimizer to determine data layout, probably more than conventional DBMS do.
• In essence, Akiban has a MySQL front-end and a storage engine back end, each running on its own hardware cluster, with each node of one cluster talking to each node of the other.
• I gather that Akiban distributes data among nodes clustered according to, in effect, object identifier. Presumably, inter-node joins are rare. But we didn’t discuss distribution, replication, or other scale-out issues in any detail. Indeed, I gathered that significant parts of all that weren’t built yet, and perhaps even not yet architected.
  

• Tom Foremski outlined the dire (at least in theory) privacy risks of geolocation services, going into a lot more detail on that point than I ever have. However, he topped that off with the odd claim that people pay toll (rather than using an electronic service) to cross the Bay Bridge because they fear being tracked, rather than for reasons of time or money.
• Oracle had an earnings conference call. Larry Dignan did a good job of covering the highlights; the gory details are on the Seeking Alpha transcript, especially pp. 3-5.  Oracle now claims to be getting lots of multi-system deals for Exadata. (But I still haven’t seen much in the way of production customers named.) ULAs, which I presume are Unlimited License Agreements, are important on the software side. Besides picking on IBM and SAP, Oracle even touted a competitive win vs. EMC, which not coincidentally seems to be working on partnering with almost every Oracle competitor it can find.
• Brian Prentice of Gartner basically accused open source of being Dotcom 2.0, in terms of dubious business models and the hype associated with same. I agree with many of his particulars, and indeed often steer vendor clients away from open source strategies. For marketing purposes, I do feel that sometimes free can be a real cool price; but open source is not the only way to be free.
• Akiban, which I wrote about a couple of days ago, seems to be building out its website. As of this writing the website is still pretty raw, with bewildering messaging, carelessly repeated paragraphs, and a notable lack of clues as to who’s in company leadership. Even so — unless I missed some of the current stuff before, the site has come a long way in a few days, so maybe there’s hope.
• Groovy Corporation, which introduced the Groovy SQL Switch just last summer, seems to be doing something different now. It’s merged into a company called uCirrus (where the u is really a mu), but uCirrus doesn’t have a meaningful website yet, whereas Groovy does. There’s stuff there about a “push data cloud,” stressing the importance of not being a DBMS, under the name Cortex, whatever that all means. Groovy seems to have an online gaming deal for Cortex with MySpace, or maybe Cortex is just the name of a specific Groovy/MySpace project.
• Mike Mooney offered a long rant on the problems with database (design) version control. He did concede that the most recent Microsoft Visual Studio might help, for those who are bought into (and can afford) the Microsoft stack. Frankly, I think that’s what views are for, updatable or otherwise. In many cases, they’ll let you build what you need, quickly and without breaking anything, and you can leave it to the DBAs to sort out database performance later.
• I just discovered Chad Stewart’s programming blog. While he’s evidently a game programmer, a lot of his comments have broader applicability.
• Chip Hazard offered a VC’s perspectives on the difficulties facing enterprise IT startups. (Hat tip to Miriam Tuerk for turning me on to him.) Although he didn’t phrase it this way, his bottom line (at least the part I agree with) is that the startup’s products have to be amazingly superior to the alternatives (big vendors or in-house).

Further edit: I’ve now posted again about Akiban, this time based on actually talking with the company.

Stealth company Akiba has renamed itself Akiban and posted what they call a “five-minute” video.* Apparently, the idea is to improve analytic query performance by denormalizing your data structure. I have no idea how this is different from denormalizing your data model in your existing DBMS, but I’ll admit to fast forwarding through the slides rather than listening to whatever the audio said.

*It’s actually 7:59 long, but who said DBMS developers should ever be believed about anything to do with schedules?

I do know one favorable thing about Akiban/Akiba, which is that Dan Weinreb is or was involved with them in some kind of angel/advisory capacity. Beyond that, all I know is that they’re in the analytic DBMS business, they’ve posted a video, they’re located in the Boston area, and they probably want people to believe that their extreme stealthiness is a sign of self-importance.

Well, there’s also what one can see on LinkedIn.