• Facebook et al. are in effect Software as a Service (SaaS) vendors, not enterprise technology users. In particular:
  • They have the technical chops to rewrite their code as  needed.
  • Unlike packaged software vendors, they’re not answerable to anybody for keeping legacy code alive after a rewrite. That makes migration a lot easier.
  • If they want to write different parts of their system on different technical underpinnings, nobody can stop them. For example …
  • …  Facebook innovated Cassandra, and is now heavily committed to HBase.
• It makes little sense to talk of Facebook’s use of “MySQL.” Better to talk of Facebook’s use of “MySQL +  memcached  + non-transparent sharding.” That said:
  • It’s hard to see why somebody today would use MySQL +  memcached  + non-transparent sharding for a new project. At least one of Couchbase or transparently-sharded MySQL is very likely a superior alternative. Other alternatives might be better yet.
  • As noted above in the example of Facebook, the many major web businesses that are using MySQL +  memcached  + non-transparent sharding for existing projects can be presumed able to migrate away from that stack as the need arises.

Continuing with that discussion of DBMS alternatives:

• If you just want to write to the memcached API anyway, why not go with Couchbase?
• If you want to go relational, why not go with MySQL? There are many alternatives for scaling or accelerating MySQL — dbShards, Schooner, Akiban, Tokutek, ScaleBase, ScaleDB, Clustrix, and Xeround come to mind quickly, so there’s a great chance that one or more will fit your use case. (And if you don’t get the choice of MySQL flavor right the first time, porting to another one shouldn’t be all THAT awful.)
• If you really, really want to go in-memory, and don’t mind writing Java stored procedures, and don’t need to do the kinds of joins it isn’t good at, but do need to do the kinds of joins it is, VoltDB could indeed be a good alternative.

And while we’re at it — going schema-free often makes a whole lot of sense. I need to write much more about the point, but for now let’s just say that I look favorably on the Big Four schema-free/NoSQL options of MongoDB, Couchbase, HBase, and Cassandra.

Last week, Mike Stonebraker insulted MySQL and Facebook’s use of it, by implication advocating VoltDB instead. Kerfuffle ensued. To the extent Mike was saying that non-transparently sharded MySQL isn’t an ideal way to do things, he’s surely right. That still leaves a lot of options for massive short-request databases, however, including transparently sharded RDBMS, scale-out in-memory DBMS (whether or not VoltDB*), and various NoSQL options. If nothing else, Couchbase would seem superior to memcached/non-transparent MySQL if you were starting a project today.

*The big problem with VoltDB, last I checked, was its reliance on Java stored procedures to get work done.

Pleasantries continued in The Register, which got an amazing-sounding quote from Mike. If The Reg is to be believed — something I wouldn’t necessarily take for granted — Mike claimed that he (i.e. VoltDB) knows how to solve the distributed join performance problem. 

So, it’s Stonebraker against the web. And the difference of option is severe. In May, at a MongoDB developer conference in San Francisco, Mongo creator Dwight Merriman told his audience there was “no way” to do distributed joins in a way that really scales. “I’m not smart enough to do distributed joins that scale horizontally, widely, and are super fast. You have to choose something else. We have no choice but to not be relational,” he said

“You can do distributed transactions, but if you do them with no loss of generality and you do them across a thousand machines, it’s not going to be that fast.”

Stonebraker says precisely the opposite, and in typical fashion, he goes right for the jugular. “I reject what Merriman says out of hand,” he tells The Register. Merriman and his company, 10gen, declined to comment for this story. But Stonebaker says words don’t matter. As much as he likes to wield his opinions, he insists the debate will be decided elsewhere. “Let the bake-off begin,” he crows.

But when last I checked, VoltDB made nowhere near that claim. And well it shouldn’t have. In the fully general case, there’s no way to ensure super distributed join performance other than by throwing lots and lots of gear at the problem. But if you do that, many alternatives are fast. More specialized cases may be a different matter — but there are many fast alternatives for those too.

I imagine there will be use cases for which VoltDB sustains a lead as the truly fastest alternative, similarly-architected competitors perhaps excepted.* But what Mike supposedly said seems quite forward-leaning when compared to technical reality.

*The canonical VoltDB use case is e-commerce in virtual goods, the point of “virtual” being that physical inventory might necessitate costlier kinds of joins.

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays.
• Human/Nontabular data — i.e., all other data generated by humans.
• Machine-Generated data.

I won’t now stand by every nuance in that post, which may differ slightly from those in my more recent posts about machine-generated data and poly-structured databases. But one general idea is hard to dispute:

Traditional database data — records of human transactional activity, referred to as “Human/Tabular data above” — will not grow as fast as Moore’s Law makes computer chips cheaper.

And that point has a straightforward corollary, namely:

It will become ever more affordable to put traditional database data entirely into RAM. 

Actually, there are numerous ways for OLTP, other short-request, and some analytic databases to wind up in RAM.

• SAP has some good ideas for how it could happen, banging transactions into what is essentially an in-memory analytic database. (I dispute SAP’s claims of transformational database technology leadership, but that doesn’t mean the underlying ideas aren’t good.)
• For those who can afford the associated technology disruption, memory-centric object-oriented DBMS could be appealing.
• Web scalability best practices commonly include keeping data in RAM (e.g., that’s pretty much the point of caching layer memcached).
• SaaS (Software as a Service) companies — such as Workday — often bring a particular tenant’s database entirely into RAM.
• QlikView highlights the benefits of doing business intelligence in RAM.
• SAS HPA makes the argument that even “big data analytics” should sometimes be done in RAM.
• I don’t have particularly favorable opinions at this time about marketing strategies or momentum at Oracle TimesTen, IBM solidDB, or VoltDB, but those examples at least serve to illustrate that memory-centric OLTP DBMS have existed for years.
• Actually, SAP has at least two good ideas, if you count Sybase as part of SAP.

And here’s the kicker: Intel told me last year that CPUs are headed to 46-bit address spaces around mid-decade. Indeed, they hired me to help figure out if that was enough.* That multiplies out to 64 terabytes of RAM on a single server, chip costs permitting. So most of what we now think of as operational databases — and many of the analytic ones too — will fit in-memory, even if they run very large businesses.

*And did so without putting the discussion under any kind of NDA.

Likely consequences of all this include:

• Legacy apps will (eventually) be consolidated and virtualized in-memory. Their underlying databases will grow so slowly that eventually the cost of putting them in RAM will be too low to worry about.
• Expensive storage systems will (continue to) be irrelevant to database processing. Databases that don’t fit in RAM will typically be big enough to require the attention of a lot of CPUs — and in those cases the DBMS software itself will handle all the storage tasks.
• Major OLTP DBMS vendors, such as Oracle, will need alternate in-memory code lines, because disk-centric architectures are sub-optimal in-memory. Well, that’s what they have those big R&D budgets for.
• SaaS vendors and web businesses may not rely on today’s major OLTP DBMS vendors. (I was going to say “won’t” rather than “may not” until I recalled the likely M&A endgame.) Traditional enterprises may blanch at migrating away from their legacy DBMS environments, but the trade-offs are different for technology companies using DBMS as subsystems.

Of course, the same trends that make data-storing chips cheaper will make data-generating chips cheaper too. So, just as there are huge amounts of machine-generated data that you’d never pay to store in RAM, the same will still be true 10 years from now; the data volumes involved will just be a lot bigger. And thus there will still be plenty of very large analytic databases using relatively cheap forms of storage, perhaps even disk.

But OLTP and other short-request processing are likely to wind up in-memory. And the same may be true for a considerable amount of analytics, especially but not only if the analytics have a low-latency requirement.

• Now we know (at least in part) why Vertica went oddly silent for a while.
• As per that same link, Vertica has >250 ordinary customers, and >70 more OEM sell-through ones.
• This is a setback for speculation about any kind of upcoming Aster/HP tie-up.
• Edit:  Forgot this one briefly — HP chairman Ray Lane was previously involved with Vertica.
• Vertica arguably is the most mature of the modern column-store DBMS — i.e., the ones that don’t have their roots in bitmaps the way Sybase and SAND do.
• HP executed really badly in data warehouse DBMS and appliances under former CEO Mark Hurd.
• Unfortunately, if you’re quickly researching Vertica, neither Gartner nor Forrester is a reliable source of detailed information.
• It would make sense for HP to acquire StreamBase too, and fold StreamBase into Vertica. Reasons include:
  • StreamBase and Vertica are aligned with each other. Both were founded by Mike Stonebraker, with overlapping groups of academic contributors. Both are in the Boston area. StreamBase and Vertica have worked together on various joint customer accounts, especially in the financial services sector.
  • Like other independent CEP vendors, StreamBase can’t or won’t accomplish much outside certain niches (mainly financial services).
  • StreamBase reports, rather credibly, that it’s doing well in its niches. While StreamBase’s success seems to include a heavy dose of professional services, that hardly would be a deal-breaker for HP.
• It would make partial sense for HP to acquire VoltDB, and fold VoltDB into Vertica.
  • VoltDB was actually spun out of Vertica, and incubated in Vertica offices. A lot of thinking has already been done about how to integrate VoltDB and Vertica.
  • VoltDB needs the help, as its strategy is not attuned to the needs of succeeding in a highly competitive, rapidly innovative marketplace.
  • VoltDB doesn’t have the kind of traction on which a big company like HP could hang an acquisition case or acquisition strategy.

Evidently Mike Olson of Cloudera is still telling the machine-generated data story, exactly as he should be. The Information Arbitrage/IA Ventures folks said something similar, focusing specifically on “sensor data” …

… and, even better, went on to say: 

Privacy is dead.
What do we consider to be the boundaries of privacy, especially with respect to items like medical data? In a data privacy-free world, should we be regulating data usage instead? How do we deal with asymmetric access to our personal data, e.g., how is it that insurance companies claim the right to our personal information?

Obviously, my answer to the second question is Yes!!!!

Also from Hadoop World — Dave Menninger, now an analyst, reports on some Hadoop metrics:

How big is “big data”? In his opening remarks, Mike shared some statistics from a survey of attendees. The average Hadoop cluster among respondents was 66 nodes and 114 terabytes of data. However there is quite a range. The largest in the survey responses was a cluster of 1,300 nodes and more than 2 petabytes of data. (Presenters from eBay blew this away, describing their production cluster of  8,500 nodes and 16 petabytes of storage.) Over 60 percent of respondents had 10 terabytes or less, and half were running 10 nodes or less.

That eBay comment was particularly interesting.

A while back, Doug Henschen noted that Netezza flagship reference Catalina Marketing is now at 2.5 petabytes. Most of that is in one 600 billion row table. Oddly, the article talks of the Netezza/SAS partnership accelerating model-building via in-database scoring (not modeling) technology. Doug also wrote of a lot of analytic DBMS replacements, including:

• Microsoft by ParAccel
• Oracle by Aster Data, IBM, Oracle Exadata, probably Netezza, and probably Hadoop
• Netezza by Greenplum
• IBM by Teradata

Carl Olofson pointed out on Twitter that DataScaler was an in-memory database technology just bought by Oracle. This inspired me to google on them, and I found a sparse DataScaler CEO blog. I link it because of an amusing juxtaposition — the second-to-last post says, in effect, “We make appliances and we recommend all these awesome technology design partners who helped us design the hardware,” while the very last post says “Designing our own hardware was a mistake.”

Fred Holahan is now VP of Marketing at VoltDB, which is a lesson to me about giving free consulting … Anyhow, Fred tells me that VoltDB has about a dozen users on their way to production, some of whom are headed to being VoltDB paying customers, some of whom are not.

It also is understood that, realistically, I can’t be expected to know and mention the very latest news for all the many products in the categories. Even so, I think this would be fine time to check just where NoSQL and HVSP adoption stand. Here is most of what I know, or links to same; it would be great if you guys would contribute additional data in the comment thread.

In the NoSQL area: 

• Back in April, the VoltDB guys told me they thought Cassandra and HBase were the two NoSQL systems with the most momentum.
• I know distressingly little about HBase adoption, but a source who may or may not wish to remain anonymous was kind enough to alert me that Twitter and StumbleUpon each have ~30 node deployments, for analytics and analytics/HVSP respectively.
• I wrote in detail on Cassandra adoption last month. News since then includes:
  • Facebook is rumored to have dropped Cassandra completely.
  • Twitter clarified that it may not be quite as lovestruck by Cassandra as before, but they’re still very close friends.
  • It’s not obvious that the Cassandra Summit unveiled a lot of new adoption stories.
• Northscale’s Membase is still in its early days.  Zynga is bought in, however, as is something called NHN Korea. (Edit: I subsequently saw NHN Korea on a prominent SEO expert’s list of the top half dozen or so search engines in the world. Who knew?)
• Basho has listed a few Riak customers. If memory serves (I haven’t spoken with Basho for a while, and some of my notes are misplaced due to some computer sloppiness), Basho has a few dozen customers in total.
• Mozilla has a 4 machine, 64 core Riak cluster in production.
• Hypertable has a few users/project sponsors, Baidu being the biggest name among them.
• I don’t really know how the MongoDB/10gen guys are doing. I think this is at least as much my fault as theirs. Anyhow, they seem to have links to a couple of folks who have written about MongoDB usage.
• NimbusDB is still in stealth mode. I’d be surprised if they had users for a while yet, since in January they didn’t yet sound as if development was very far underway. (Actually, I forget whether NimbusDB is supposed to be SQL-based or not.)

Among the SQL or SQL-friendly guys:

• Clustrix says it has a few production users, some big-name, but is not disclosing them yet.
• dbShards has around 6 customers, including Facebook. (Facebook may outpace even Twitter and Zynga in using the most products mentioned in this post.)
• As of May, VoltDB had one paying customer, plus 150 beta customers who weren’t in production yet.
• Akiban says they’ll get me up to speed on Thursday.
• ScaleDB seems to be pedaling along in perennial beta. Whether ScaleDB has any actual beta users is less clear. On the plus side, checking that out uncovered a pretty funny April Fool blog post.
• Groovy Corporation seems to have disappeared, or morphed into something called uCirrus, or something like that.

*Who, in Todd’s apt description, is “the sword wielding Johnny Appleseed of the database world”.

Todd wrote:

What matters to VoltDB is: speed at scale, speed at scale, speed at scale, SQL, and ACID. If that matches your priorities then you’ll probably be happy. Otherwise, as you’ll see, everything is sacrificed for speed at scale and what is sacrificed is often ease of use, generality, and error checking. It’s likely we’ll see ease of use improve over time, but for now it looks like rough going, unless of course, you are a going for speed at scale.

Indeed.

Todd’s list of interesting VoltDB features is also pretty good, namely

Main-memory storage.

Run transactions to completion –single threaded –in timestamp order. 

Replicas.

Tables are partitioned across multiple servers.

Stored procedures, written in Java, are the unit of transaction.

A limited subset of SQL ’99 is supported.

Design a schema and workflow to use single-sited procedures.

Challenging operations model.

No WAN support.

OLAP is purposefully kept separate.

• VoltDB is based on the H-Store technology, which I wrote about in February, 2009. Most of what I said about H-Store then applies to VoltDB today.
• VoltDB is a no-apologies ACID relational DBMS, which runs entirely in RAM.
• VoltDB has rather limited SQL. (One example: VoltDB can’t do SUMs in SQL.) However, VoltDB guy Tim Callaghan (Mark Callaghan’s lesser-known but nonetheless smart brother) asserts that if you code up the missing functionality, it’s almost as fast as if it were present in the DBMS to begin with, because there’s no added I/O from the handoff between the DBMS and the procedural code. (The data’s in RAM one way or the other.)
• VoltDB’s Big Conceptual Performance Story is that it does away with most locks, latches, logs, etc., and also most context switching.
• In particular, you’re supposed to partition your data and architect your application so that most transactions execute on a single core. When you can do that, you get VoltDB’s performance benefits. To the extent you can’t, you’re in two-phase-commit performance land. (More precisely, you’re doing 2PC for multi-core writes, which is surely a major reason that multi-core reads are a lot faster in VoltDB than multi-core writes.)
• VoltDB has a little less than one DBMS thread per core. When the data partitioning works as it should, you execute a complete transaction in that single thread. Poof. No context switching.
• A transaction in VoltDB is a Java stored procedure. (The early idea of Ruby on Rails in lieu of the Java/SQL combo didn’t hold up performance-wise.)
• Solid-state memory is not a viable alternative to RAM for VoltDB. Too slow.
• Instead, VoltDB lets you snapshot data to disk at tunable intervals. “Continuous” is one of the options, wherein a new snapshot starts being made as soon as the last one completes.
• In addition, VoltDB will also spool a kind of transaction log to the target of your choice. (Obvious choice: An analytic DBMS such as Vertica, but there’s no such connectivity partnership actually in place at this time.)

I should also note that when Tim Callaghan described architectural options to get around 2PC performance issues, they sounded a lot like eventual consistency. Maybe tunable RYW consistency isn’t in the cards, but at least there’s a NoSQL-like possibility with VoltDB.

VoltDB’s open source strategy is:

• VoltDB will be open sourced.
• Community VoltDB will be GPLed. Professional Edition VoltDB has a non-GPL license.
• The VoltDB Professional Edition won’t start out with features beyond the Community Edition ones, but will gain such later on. I didn’t get the sense the plans for those features were completely baked yet, but ideas mentioned included:
  • Management/monitoring tools.
  • Integration with expense closed-source enterprise software products, such as ones in the management/monitoring area.
  • Yet more “extreme”/edge-case performance.
• Before VoltDB decided for sure that it wasn’t selling licenses, it sold a license to Getco, which also seems to be an investor in the company.

VoltDB had a beta test with about 150 participants. None is in production yet, although at least a few are clearly headed there. Most VoltDB beta testers are in some kind of online business, with a particular concentration in everybody’s new favorite market, online gaming. Most of the rest are in investment/trading — a major target market for at least three different Mike Stonebraker companies — and a few are in telecom. VoltDB assures me that some of the beta users are companies one actually has heard of before, but VoltDB is not in a position to name any of those.

VoltDB is not ideally suited for a classic order management system, since you’d want to partition both on CustomerID and SKU, the latter because you’d constantly updating inventory stock levels. However, this argument doesn’t apply in the case of virtual goods. Virtual goods that are sold for real money — and hence need ACID levels of transaction integrity — are thus a clear target market for VoltDB. (The example that came up was in, you guessed it, online gaming.) The other interesting use case that Tim highlighted was low-latency analytics/ELT. For reasons I didn’t totally grasp, Tim likes to call this “Stateful ELT.” (Given that the data goes into the VoltDB database before much else happens to it, I’m pretty sure I heard “ELT” correctly. But I guess I might have been mishearing “ETL”.)

VoltDB company highlights include:

• VoltDB has about a dozen employees, all but two of whom are technical. (However, I’m not sure they’re counting Andy Ellicott against the two. But then, last I heard he wasn’t full time at VoltDB.)
• VoltDB’s venture funding status is, if I may paraphrase, “Mumble mumble.”
• Although long separate from Vertica, VoltDB is still located in Vertica’s offices.

Nonetheless, OLTP/general-purpose data management startup activity has recently picked up, targeting what I see as some very real opportunities and needs. So as a jumping-off point for further writing, I thought it might be interesting to collect a few observations about the market in one place.  These include:

• Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.
• By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value.
• Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.
• It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.
• The era of silicon-centric relational DBMS is coming.
• The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds.
• Users’ instance on “free” could be a major problem for OLTP DBMS innovation.

I shall explain.

Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.

• OLTP applications are more complex than analytic ones, and hence more tightly wired into particular brands of DBMS. For example, third-party packaged OLTP applications are typically portable among only a few brands of DBMS. But third-party business intelligence tools, and the BI “applications” built in them, are more easily and widely portable.
• Specific technical observations such as “OLTP apps tend to use stored procedures, which are DBMS-specific” or “OLTP apps tend to have lots and lots of tables” serve to underscore the first point.
• An enterprise’s highest-value data is commonly the financial stuff handled by its core OLTP systems, so those are the last things they want to mess around with just to get some cost savings. Security, high availability, and so on are major considerations that can outweigh cost.

By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value. Indeed, “OLTP” is often a misnomer, which is why I tend to go with “general-purpose” or some similarly wishy-washy phrase instead.

• In theory, this is a ripe area for what I’ve called mid-range DBMS.
• The big brand vendors try hard to keep as many of those databases for themselves as they can. Enterprise-wide license pricing helps. Going forward, so will virtualization/consolidation strategies, such as Oracle’s Exadata-centric approach.
• A variety of mid-range DBMS alternatives beyond the big brands have technical merit, at least in some cases and configurations – MySQL, PostgreSQL, Intersystems Cache’, and so on.
• The only such mid-range DBMS alternative with much large enterprise business momentum, however, appears to be MySQL.

“General-purpose” might be a better term than “OLTP” anyway.

• I don’t have a link, but it’s widely agreed that over half of the processing on an “OLTP” enterprise app is commonly reporting and so on.
• “Operational BI” is progressing by fits and starts, but it is progressing.
• Anything customer-facing — web-based, call center, or otherwise — is likely to include a heavy dose of “real-time” analytic optimization.

Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.

• VoltDB is the main exception that jumps to mind.
• This isn’t true in the analytic DBMS area, where Netezza, Greenplum, Aster, Vertica and others started from PostgreSQL’s code, APIs, or both.

It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.

• MySQL is a limited product without all the third-party storage engines that are being developed.
• Oracle’s promise of MySQL good behavior has an expiration date.
• None of the MySQL front-end alternatives are remotely mature yet.

The era of silicon-centric relational DBMS is coming.

• I think “silicon” means “solid-state memory” as much as or more than it means “RAM,” but that’s not yet certain.
• What is pretty certain is that, thanks to Moore’s Law, some kind of silicon will increasingly replace disk.
• Oracle’s increasingly Flash-centric story is a challenge to everybody.
• RAM-centric VoltDB will launch fairly soon. (By the way, while VoltDB still has a lot in common with H-Store, they’re not exactly the same thing. And H-Store research is progressing too.)
• RethinkDB is being developed, focused directly on solid-state memory. Based on the sparse information available online, RethinkDB sounds somewhat like a dumbed-down H-Store.
• New disk-based vendors may never optimize their use of disk, instead targeting a solid-state future. (E.g., I think Akiban should and quite well might follow this path.)

The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds. We hear that from the NoSQL guys all the time. But I also just heard it from Akiban.

Users’ instance on “free” could be a major problem for OLTP DBMS innovation. Vendors of new OLTP data management technologies often feel obligated to open source their products, notwithstanding the historical lack of revenue in the open source OLTP DBMS market. As just one of many examples, Nova Spivack wrote:

I have recently seen some new graph data storage products that may provide the levels of scale and performance needed, but pricing has not been determined yet. In short, storage and retrieval of semantic graph datasets is a big unsolved challenge that is holding back the entire industry. We need federated database systems that can handle hundreds of billions to trillions of triples under high load conditions, in the cloud, on commodity hardware and open source software. Only then will it be affordable to make semantic applications and services at Web-scale.

I hear similar things from other startups, who evidently believe they need and/or are entitled to enjoy sophisticated, high-performance, zero-cost, specialized database management technology.