• Dwight Merriman of 10gen (MongoDB)
• Damien Katz of Couchio (CouchDB)
• Matt Pfeil of Riptano (Cassandra)
• Todd Lipcon of Cloudera (HBase committer)
• Tony Falco of Basho (Riak)
• John Busch of Schooner
• Ori Herrnstadt of Akiban

By no means do I have time to do these conversations justice, in terms of giving them the write-ups and/or immediate follow-up that they deserve. Indeed, I’ll leave for vacation Saturday morning with my 2000-word NoSQL article still unwritten. So I’ll dump as many observations as I can into one or a few posts now, and play catch-up later as circumstances allow.

In no particular order:

• A number of NoSQL offerings have had more uptake to date than most of the scale-out SQL offerings have.
• “Document-oriented” NoSQL projects CouchDB and MongoDB have probably had the most users get into production, but perhaps for pretty small systems.
• Cassandra and Hbase — the column-group-architecture guys — have probably had the most bang-in-lots-of-writes HVSP production uptake.*
• I didn’t talk customer count with Schooner, but the decently-stocked Schooner customer page suggests Schooner may be something of an exception to these generalities.
• A lot of these companies are in the low-to-mid-teens of employees.
• The SQL-oriented companies, despite having fewer or no customers, often seem to have more money. (One reason I get the impression SQL guys have more money is, frankly, that more of them are talking about engaging my services.)
  • Schooner cites $20 million in VC.
  • Clustrix cites a figure close to that.
  • Basho cites $10 million, plus a new round of $1.5 or $2 or $2.5 million. The new round is at a  lowered valuation.
  • That same site says Tokutek finally was able to raise some VC. Congrats!
• It’s only a two-company trend, but I was pleased to hear that both 10gen/MongoDB and Akiban were seeing Drupal as a major use case or potential use case. No word on rescuing WordPress from its MySQL implementation, alas, but it seems that a Drupal site typically has 40-200+ tables, while a WordPress one has 10ish.
• Another trend I think I’m seeing is serious object-oriented apps banging things straight into a simple back end. Workday is a huge example of that. Akiban hopes to do something similar with Hibernate.
• Stability and maturity are still issues for many of these products. E.g., HBase isn’t even in Release 1.0 yet. Ditto Cassandra, and surely many of the others. Unsurprisingly, making Cassandra stable is still a challenge.

*As is common for terms I suggest, the “HVSP” name is not getting any traction. What do you think of Marton Trencseni’s suggestion of OLRP, for OnLine Request Processing?

One thing that makes following this area interesting is that so many projects are open source, leading there to be a lot of information in the wild. I hardly have time to read the mailing list for each project; but the people I talk with do, and often they may sorta kinda remember something somebody else posted one or several months back. As just one example, the mailing lists are said to confirm:

• Contrary to rumor, Facebook hasn’t moved in-box search off of Cassandra.
• Apparently, however, it’s true that Cassandra inventor Facebook has stopped working on Cassandra, and Facebook’s core Cassandra developers have shifted over to HBase.

Also, figuring out usage of open source software can be … interesting.

• People who use open source software don’t have to reveal themselves, as there’s no purchase transaction to kick things off.
• On the other hand, if they’re serious enough in their use, they often do.
  • There are two main ways to get tech support for open source software — the community or a company that sells support — and both ways let the main support-selling company know that one is a user.
  • Some folks even add themselves to open lists of users, for example these rather long lists for HBase and CouchDB.
  • Or they show up at conferences. For example, two tweets from Riptano founder Jonathan Ellis suggest at least 30 production Cassandra users were represented at a recent event. That’s more detail than his colleague Matt Pfeil wanted to give me when talked.

OK. This post has gotten pretty long, even without me saying anything resembling an overview of any of the seven companies I listed up top, or of their products’ adoption. So I’ll just publish this now, and edit in links below to follow-on posts if and when they become available.

• Unemployed, underemployed, or otherwise desirous of having more commercial skills.
• Not a programmer, but desirous of having some technical skills.
• Astute enough to realize s/he will never be a serious techie.

I generally have two models in mind when guiding such a person:

• Analytics/business intelligence/stats.
• Website building.

Those are both useful skill sets for people who aren’t full-time techies, the first perhaps best for those who are more quantitative and big-company-friendly, the second perhaps better for the creative and/or rebellious types.

So what SPECIFICALLY should one guide them to do? My initial thoughts include:

• Learning Java is overkill for most of these people.
• Learning C++ is overkill for ALL of these people. If you’re not out to be a hardcore engineer, the “advantages” of C++ over Java are pointless.
• They all should learn some SQL.
• MySQL is the most accessible DBMS against which to learn SQL. They should download a (free) copy and install it on their PC.
• But I have no idea which books or websites they should go to to learn about SQL.
• While at first blush it sounds like overkill, downloading and installing the free version of Microstrategy 9 is a good way to learn about BI and also the analytic side of SQL.
• The first thing you learn in an app dev tool used to be and probably still is how to do a master-detail form. That would cover the other side of learning SQL. But what would be a good choice of tool? (Preferably free, as building serious OLTP apps is probably not what these people will want to do.)
• One idea I had is that the website-oriented ones should learn how to modify WordPress, by which I really mean modifying WordPress themes. That would involve learning PHP, SQL, and HTML/CSS, which seems like a great place to start.
• But I have no idea which books or websites they should go to to learn about PHP.
• I also have no idea which books or websites they should go to to learn about CSS — or for that matter even basic HTML.
• If they want to take the analytics route, I assume R is the way to go. Thoughts?
• Python isn’t the ideal language for much of anything, but it’s an easily accessible “first language”. Umm, is that a good way to go, or would PHP be a better choice?
• Any other ideas?

For anybody who pitches in — thanks!! I hope to get enough useful answers so as to keep editing this post with people’s ideas.

Edit: Suggestions have started to come in on Twitter. A couple of folks are saying that HTML is a good place to start. Hard to argue with that, although it’s hardly where one should finish. There also was a vote for Yahoo YQL, and of course for a vendor’s own product.

Some great points are in the comments below, including the idea that you should pick an actual, fun, small project to build to get you started. (A site built in WordPress or Mambo would be a pretty obvious choice for such a project, come to think of it.)

• Despite heavy emphasis on the word “sharding,” dbShards’s scale-out is transparent to the application programmer. E.g., in dbShards + MySQL, the APIs are more or less the same ones you’d expect for MySQL (JDBC, etc.)
• If the DBMS underneath is ACID-compliant (e.g., MySQL + InnoDB), then the dbShards version is ACID-compliant too.
• Beyond those basics, I forgot to check the fine details of dbShards’ MySQL (or PostgreSQL) syntax support. Todd Hoff, however, did not forget.
• dbShards keeps copies of each shard on two different servers, via asynchronous log-shipping. This allows for failover in both planned and unplanned outages.
• dbShards wants you to distribute big tables among shards via a “shard key,” which is a lot like the distribution key in MPP analytic DBMS. You’re encouraged to replicate small, low-update-volume tables across each shard.
• Cory says that dbShards has good join performance when – you guessed it! – everything being joined is co-located shard-by-shard, because the tables were distributed on the same shard key and/or replicated across each shard. Cory can’t imagine why you’d want to do an inner join under any other circumstances.
• The basic dbShards query execution model is: A query comes in; it’s parsed; a shard key is automagically detected (one hopes); the “global configuration file” is checked to see which shard to ship the work off too. I forgot to ask whether lookup was done via a hash table (the obvious guess) or something else. The programmer can put hints in the code comments to direct the sharding, but Cory asserts those aren’t needed very often.
• Cory says that insert performance with dbShards + MySQL + InnoDB is 1500-3000 inserts per shard per second, scaling almost linearly with the number of shards. I forgot to ask how many shards this had been tested for.
• If you want blazing dbShards performance, Cory’s base-case figure is 25 gigabytes of data per node, so that the most commonly used indexes can camp out in memory. (I forgot to ask what kind of hardware he was assuming per node.) This is if you’re going to be doing joins or aggregrations. If it’s just single-row inserts and updates, or if your performance requirements are lower, you can go with 10X that figure.
• Cory tells stories wherein going from an unsharded database to 4 or so shards took database re-indexing time down 50X or more. Apparently, such tasks can be exponential or even super-exponential with database size over InnoDB. (That said, I’d be surprised if all large InnoDB users suffered from that problem to the same degree.)
• dbShards’ customer workloads are all >= 50% reads. This is reflective of dbShards’ design priorities.
• As long as it can be in charge, dbShards is happy to interface to whatever kind of database backup software you want to use on a node by node basis. (dbShards wants to drive your backup software for you so that it can be sure the replicas are handled properly.)
• It’s “fairly common” for dbShards to be paired with memcached. I forgot to ask whether memcached typically lived on its own pool of servers, or on the same pool that runs dbShards.
• Future DBMS options under consideration for dbShards include Oracle and (unspecified) in-memory.

Business highlights for CodeFutures and dbShards include:

• dbShards’ price is $5000/server/year, including support and OEMed MySQL, with stated quantity discounts up to 40%.
• dbShards cloud pricing is different (on a usage basis).
• dbShards has 6 or so customers, half each on-premises and in the cloud. One of them is Facebook. (Those “100s” of customers mentioned on the dbShards website are for a fairly unrelated product.)
• CodeFutures has been at this 2 ½ years or so. There is no venture capital in the company.
• Early deals dbShards deals have evidently involved a fair amount of professional services.
• Counting contractors, Code Futures has 10-12 people, which has been as high as 15.
• Target dbShards customers are as you’d expect. Cory says he’s actually been more successful getting early-adopted money out of Web companies than Wall Street firms.
• There are a couple of dbShards PostgreSQL customers for greenfield applications. Most dbShards customers and prospects, however, are looking to scale out existing apps.
• Despite its connection to open source DBMS, there’s nothing open source about dbShards itself.

A consensus seems to have emerged that Thesis had actually copied beyond-fair-use amounts of WordPress code, which if true was Game Over. Beyond that, however, both sides of the strongly-viral-GPL debate scored some points.

• Public pressure, FUD, etc. in favor of the GPL clearly were successful.
• Mark Aquith carefully explained  how tightly integrated WordPress and WordPress themes are. So we don’t necessarily have much of a precedent for more hands-off integration.
• Previous precedents were dredged up on both sides. More on that below.
• Once again, the claims about the GPL were not tested in court. Various open source developers and lawyers have stated their opinions as to what their rights are, but the courts were not given the opportunity in this case to (dis)agree on the core issue …
• … which everybody seems to be correctly agreeing is: “When is one software program a ‘derivative work’ of another one, in the copyright-law sense of ‘derivative work’?”

The pro-GPL argument on that last point would probably boil down, colloquially, to “Well, is it one program or two? If it’s one, then it clearly is derivative from the big part that’s copyrighted under the GPL. If it’s two, then the creator of the second one is home free — but c’mon, now, it’s really just one.” That, in turn, would be supported by Dan Weinreb’s point from a previous MySQL storage engine/GPL comment thread (which also has a lot of applicability to the WordPress theme case) – you really don’t want to make the user do two installs, so you really do want to include the GPLed code in your package. And by the way — so far the product packaging by the MySQL storage engine vendors* is in line with Dan’s observation.

*Infobright, Akiban, Tokutek, Calpont, et al.

One point I haven’t seen discussed much yet is this:

Suppose a MySQL storage engine vendor integrated with a forked, GPLed MySQL, and then didn’t obey the GPL. Who would have standing to sue them? It’s obvious that the developers of the forked MySQL would. But it’s not at all obvious that Oracle would. A derivative of a derivative of a copyrighted work is NOT necessarily a derivative of the original. (Think about it.) Unless Oracle could prove that the MySQL storage engine really did happen to be a derivative of MySQL Classic, I don’t know why Boogeyman Oracle would have standing to sue.

Finally, those precedents.

• A couple of comments on my earlier post point out that the Linux community tends to be pretty tolerant of proprietary code that links tightly into Linux. Oh, they may not like — but in most cases they neither do sue nor believe they successfully can.
• Wikipedia cites some cases in which the GPL has been successfully enforced.
• One of the earliest GPL controversies was over what sounds like a MySQL storage engine — Progress Software’s NuSphere, developed in connection with MySQL AB. Litigation ensued, and before the case was settled, the judge wrote “After hearing, MySQL seems to have the better argument here, but the matter is one of fair dispute.“

Now a related can of worms has been opened in a related technology area — WordPress and WordPress themes. Since many bloggers use WordPress, this has gotten a lot of attention, and some interesting new insights have emerged.

Um, in case you didn’t know: WordPress is the software that runs blogs such as this, and it’s a GPLed open source project. However, the user interface — look, feel, and behavior alike — are determined by separate themes, that one usually gets from third parties (WordPress ships with a a few default choices).

It started when Matt Mullenweg went after the makers of an unfree theme Thesis, and wielding a legal opinion from the Software Freedom Law Center. The gist of the SFLC’s argument seems to be

They are derivative of WordPress because every part of them is determined by the content of the WordPress functions they call. As works of authorship, they are designed only to be combined with WordPress into a larger work.

And of course the point of the GPL is that if you create a derivative work of something GPLed, you have to GPL it yourself.

However, Perpetual Beta pointed out that, under the rules of copyright law as expressed in a court case known as Galoob, depending on another program does not make something a derivative work. This is actually blindingly obvious, as in the example of any program that runs on top of an operating system. Or for more examples see Dave Winer on the point.

Perpetual Beta further argued that, even if it were a derivative work, fair use would let one copy it anyway. I.e., if you’re engaging in “fair use,” you’re entitled to do what otherwise would be a copyright violation. Good point. The GPL license says in effect “You only are allowed to use this material (in certain ways) if you do as we say about your own work,” so that is defeated if the Fair Use Doctrine lets you say “Um, actually, I’m using this without your permission, so buzz off.”

GPL advocates can pontificate all they want about certain uses of GPLed code violating their license terms. But if either of these arguments holds up — and it looks to me like both do — a program that invokes GPLed code is not subject to the GPL just based on those invocations. And that, in turn, would more than imply that MySQL storage engine vendors could use GPLed MySQL-compatible front-ends without being under any GPL obligation themselves.

• Performance and bottleneck cleanup.
• “Omigod, you mean you didn’t have that feature before?” cleanup.

That said, the traditional release for cleaning up the last huge gaps in an analytic DBMS product seems have become 4.0; recent examples include Aster Data, Vertica and Greenplum. Infobright seems on track to be another example of that rule.

Ack. Now that I’ve said that, other vendors are going to be tempted to accelerate their numbering so as to reach the 4.0 mark sooner …

A lot of Infobright performance enhancements are in the vein “We used to rely on generic MySQL for that, but now we do it ourselves, and it works a lot better.” Examples include: 

• Infobright now does DELETEs all at once, vs. the previous row-by-row way. This makes DELETE performance similar to SELECT performance, when previously there was a big difference.
• Ditto, if I understood correctly, INSERTs and UPDATEs.
• Each release, Infobright covers more SQL functionality itself and passes less through to the generic MySQL engine.
• UTF-8 Unicode data can now be loaded via Infobright’s parallel loader. Previously, you had to use MySQL’s load.

Infobright has also added workload management in 3.4, and this is intertwined with multicore parallelization, apparently because the workload manager decides when a query should use multiple cores to execute. Infobright further says that multi-user INSERT performance has increased a lot more than single-user, but I have forgotten why that is.

Infobright now streams data back to the client faster. E.g., unless there’s some good reason not to, partial query results are pipelined back as they become available.* Finally, loading data no longer locks tables from being read (in my notes I wasn’t sure whether that was a current or future feature, but Infobright’s marketing seems to indicate it’s current). For some reason, Infobright is positioning this as a major, innovative feature.

*A good reason not to do this would be an aggregate that requires full materialization of the table before Infobright can carry it out.

]]> http://www.dbms2.com/2010/06/27/infobright-release-3-4/feed/ 6 http://www.dbms2.com/2010/05/12/the-clustrix-story/ http://www.dbms2.com/2010/05/12/the-clustrix-story/#comments Wed, 12 May 2010 08:53:48 +0000 Curt Monash http://www.dbms2.com/?p=2096 After my recent post, the Clustrix guys raised their hands and briefed me. Takeaways included:   

• Nothing in my original short post about Clustrix was actually incorrect.
• Clustrix plans to reveal actual production “name-brand” customers soon.
• The name of Clustrix’s software, or at least the guts thereof, is Sierra.
• Clustrix’s products have actually been in general availability since last quarter, with some versions at customer sites for 2 years. Development started 3 ½ years ago.
• Clustrix says its technology is for OLTP systems, which it calls “non-batch/non-analytic,” with mixed read/write workloads. All Clustrix’s example target markets are “internet verticals,” such as photo sharing, gaming, social media, e-commerce, etc.
• Clustrix’s heart is in SQL, as is most of its customer base. Clustrix Sierra’s key-value-store option has little or no performance advantage over Clustrix Sierra’s SQL option, nor any other advantage over SQL that came up in discussion.
• Clustrix Sierra is “wire-compatible” with MySQL, but doesn’t use MySQL code; Clustrix wrote all the code itself.
• Clustrix asserts that Clustrix Sierra supports the “vast majority” of MySQL features. Examples of MySQL features Clustrix doesn’t support at this time are full-text search and geospatial indexing.
• Indeed, Clustrix claims Clustrix Sierra can be used to replace MySQL with few or zero changes to existing applications.
• I specifically asked about referential integrity, which has a poor performance reputation in MySQL. Besides saying they supported it, Clustrix said that some customers actually use referential integrity in some of their less active tables.
• Clustrix Sierra is fully ACID-compliant, with no eventual consistency or RYW consistency story. The default number of copies of each datum is two, and they’re kept consistent via two-phase commit.
• Clustrix Sierra is fully parallel, with no “head” node. I forgot to ask how it was determined which queries would be addressed to and/or controlled by which nodes, but I presume there’s some sort of a load-balancing scheme.
• Clustrix says that because Clustrix Sierra uses MVCC (Multi-Version Concurrency Control), and thus reads and writes don’t block each other, global locks aren’t a major issue. (They’re rare or short or something – I have trouble seeing why they would be non-existent.)
• Clustrix says there’s a second class of locks and latches that are purely local and short-lived, for B-tree indexes and the like. (I didn’t drill down into those either.) I guess this means Clustrix Sierra is B-tree-centric, which makes sense for an OLTP-oriented system.
• Clustrix Sierra distributes data among nodes via consistent hashing (default), range partitioning, or “full distribution”(i.e., coping a – presumably small – table to each node). The choice of distribution plans is manual now; more automation is a future feature.
• Clustrix Sierra’s CBO (Cost-Based Optimizer) is, as one would hope, distribution-aware.
• Clustrix Sierra compiles query fragments and ships them off to the relevant nodes. A fragment might contain both instructions for SQL to be executed locally and for where data is to be sent next.
• Clustrix says that Clustrix Sierra does data migration and redistribution (e.g., when you add a node) transparently online, and further says that in practice this doesn’t cause a performance hit.
• As for Clustrix hardware:
  • Clustrix makes Type I appliances.
  • A Clustrix node contains 2 quad-core chips, 32 gigs of RAM, and 7 160 GB solid-state drives.
  • Specifically, Clustrix is using Intel SSDs, with a SAS interface.
  • Clustrix says solid-state memory isn’t really essential to the product design; it’s just cheap in terms of $/IOPS (I/O Per Second).
• A minimum Clustrix configuration is 3 nodes, for redundancy. After that you can add nodes one at a time. Clustrix says it built a 20-node system in-house, leading me to suspect that customers don’t have anything bigger than 20 nodes either.
• That 20-node Clustrix system was tested to show near-linear scalability. (In discussing this, Clustrix tends to forget to use the word “near”.)
• Clustrix has partnered with somebody to provide global 4-hour-response support. As of now Clustrix seems to be active mainly in North America and Europe.
• Clustrix is formed from the combination of two startups, which I’ve heard elsewhere were called Clustrix and Sprout. Exactly when the combination happened sounds a little different depending on who’s telling the story (one version has the predecessors still being separate well into 2008, but Clustrix implies the combination happened pretty much on Day 1).

Nonetheless, OLTP/general-purpose data management startup activity has recently picked up, targeting what I see as some very real opportunities and needs. So as a jumping-off point for further writing, I thought it might be interesting to collect a few observations about the market in one place.  These include:

• Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.
• By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value.
• Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.
• It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.
• The era of silicon-centric relational DBMS is coming.
• The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds.
• Users’ instance on “free” could be a major problem for OLTP DBMS innovation.

I shall explain.

Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.

• OLTP applications are more complex than analytic ones, and hence more tightly wired into particular brands of DBMS. For example, third-party packaged OLTP applications are typically portable among only a few brands of DBMS. But third-party business intelligence tools, and the BI “applications” built in them, are more easily and widely portable.
• Specific technical observations such as “OLTP apps tend to use stored procedures, which are DBMS-specific” or “OLTP apps tend to have lots and lots of tables” serve to underscore the first point.
• An enterprise’s highest-value data is commonly the financial stuff handled by its core OLTP systems, so those are the last things they want to mess around with just to get some cost savings. Security, high availability, and so on are major considerations that can outweigh cost.

By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value. Indeed, “OLTP” is often a misnomer, which is why I tend to go with “general-purpose” or some similarly wishy-washy phrase instead.

• In theory, this is a ripe area for what I’ve called mid-range DBMS.
• The big brand vendors try hard to keep as many of those databases for themselves as they can. Enterprise-wide license pricing helps. Going forward, so will virtualization/consolidation strategies, such as Oracle’s Exadata-centric approach.
• A variety of mid-range DBMS alternatives beyond the big brands have technical merit, at least in some cases and configurations – MySQL, PostgreSQL, Intersystems Cache’, and so on.
• The only such mid-range DBMS alternative with much large enterprise business momentum, however, appears to be MySQL.

“General-purpose” might be a better term than “OLTP” anyway.

• I don’t have a link, but it’s widely agreed that over half of the processing on an “OLTP” enterprise app is commonly reporting and so on.
• “Operational BI” is progressing by fits and starts, but it is progressing.
• Anything customer-facing — web-based, call center, or otherwise — is likely to include a heavy dose of “real-time” analytic optimization.

Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.

• VoltDB is the main exception that jumps to mind.
• This isn’t true in the analytic DBMS area, where Netezza, Greenplum, Aster, Vertica and others started from PostgreSQL’s code, APIs, or both.

It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.

• MySQL is a limited product without all the third-party storage engines that are being developed.
• Oracle’s promise of MySQL good behavior has an expiration date.
• None of the MySQL front-end alternatives are remotely mature yet.

The era of silicon-centric relational DBMS is coming.

• I think “silicon” means “solid-state memory” as much as or more than it means “RAM,” but that’s not yet certain.
• What is pretty certain is that, thanks to Moore’s Law, some kind of silicon will increasingly replace disk.
• Oracle’s increasingly Flash-centric story is a challenge to everybody.
• RAM-centric VoltDB will launch fairly soon. (By the way, while VoltDB still has a lot in common with H-Store, they’re not exactly the same thing. And H-Store research is progressing too.)
• RethinkDB is being developed, focused directly on solid-state memory. Based on the sparse information available online, RethinkDB sounds somewhat like a dumbed-down H-Store.
• New disk-based vendors may never optimize their use of disk, instead targeting a solid-state future. (E.g., I think Akiban should and quite well might follow this path.)

The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds. We hear that from the NoSQL guys all the time. But I also just heard it from Akiban.

Users’ instance on “free” could be a major problem for OLTP DBMS innovation. Vendors of new OLTP data management technologies often feel obligated to open source their products, notwithstanding the historical lack of revenue in the open source OLTP DBMS market. As just one of many examples, Nova Spivack wrote:

I have recently seen some new graph data storage products that may provide the levels of scale and performance needed, but pricing has not been determined yet. In short, storage and retrieval of semantic graph datasets is a big unsolved challenge that is holding back the entire industry. We need federated database systems that can handle hundreds of billions to trillions of triples under high load conditions, in the cloud, on commodity hardware and open source software. Only then will it be affordable to make semantic applications and services at Web-scale.

I hear similar things from other startups, who evidently believe they need and/or are entitled to enjoy sophisticated, high-performance, zero-cost, specialized database management technology.

• Akiban is basically in the business of making OLTP (OnLine Transaction Processing) DBMS.
• That said, Akiban does not necessarily aspire to offer the DBMS that has the best update efficiency or throughput. In particular, the Akiban DBMS stores every datum twice, even before replication (and indexes) are taken into account.
• Akiban wants to store everything as third normal form relational databases. I didn’t ask whether 3NF is a hard requirement, a Really Good Idea if you want Akiban to run fast (that’s the one I’d guess), or merely a general design assumption.
• Akiban characterizes its core differentiators/value proposition as being:
  • Scale-out
  • No need to pay the traditional cost of joins
• Thus, Akiban is telling something like a NoSQL story.
• However, Akiban offers SQL.
• Specifically, Akiban offers SQL through a MySQL front end. However, the choice of front-end could change (Drizzle?), and non-relational front-ends (object?)* could eventually also be offered.
• Akiban’s first target market is SaaS providers, specifically ones that have true multitenancy issues. More generally, Akiban is pursuing cloud/private cloud applications with lots of tables. (Ori talks of a few thousand tables as being a small number.) At least at first, Akiban is conceding the market for huge-volume, scale-out, no-expensive-join web databases to the NoSQL contenders.
• Akiban has been in prototyping/development of some kind for several years. However, Akiban got its first angel funding early last year and its first venture funding late in 2009, so development only ramped up recently.
• Ori tells a version of the rather common “Everything I need to know in life I learned in the Israeli Army, and now I’m commercializing it” story. However, I didn’t get the sense that Akiban is necessarily a direct extension of a specific Israeli military project.

* A lot of Boston-area DBMS developers have significant non-relational experience. E.g., Jack Orenstein was an Object Design founder, and Peter Beaman used to work for Intersystems, both object-oriented DBMS vendors.

Akiban technical highlights include:

• Somewhat confusingly, Akiban databases are divided into “groups” of tables. The point of Akiban groups is:
  • Many-to-many relationships exist only within Akiban groups, not among tables in different groups.
  • Tables within Akiban groups are kind of pre-joined; more precisely, data is organized physically in a way that anticipates joins.
  • Thus, most Akiban joins can be executed without the cost of traditional join algorithms.
• One copy of the data Akiban stores is, in effect, clustered by object. E.g., a customer and her orders are stored together, or a patient and the records of her doctor visits. That’s how Akiban anticipates most joins.
• The other copy of Akiban data is stored in columns (I’m not sure if this part is strictly columnar or more hybrid row/column), which are ordered consistently. In particular, they’re in an order dictated by the organization of the other copy of the data, whatever that means. Akiban’s goal is for this copy of the data to support reporting, operational BI, etc.
• Akiban relies heavily on its optimizer to determine data layout, probably more than conventional DBMS do.
• In essence, Akiban has a MySQL front-end and a storage engine back end, each running on its own hardware cluster, with each node of one cluster talking to each node of the other.
• I gather that Akiban distributes data among nodes clustered according to, in effect, object identifier. Presumably, inter-node joins are rare. But we didn’t discuss distribution, replication, or other scale-out issues in any detail. Indeed, I gathered that significant parts of all that weren’t built yet, and perhaps even not yet architected.
  

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.