• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

1. Continued Availability of Storage Engine APIs. Oracle shall maintain and periodically enhance MySQL’s Pluggable Storage Engine Architecture to allow users the flexibility to choose from a portfolio of native and third party supplied storage engines.

MySQL’s Pluggable Storage Engine Architecture shall mean MySQL’s current practice of using, publicly-available, documented application programming interfaces to allow storage engine vendors to “plug” into the MySQL database server. Documentation shall be consistent with the documentation currently provided by Sun.

Well, duh.

2. Non-assertion. As copyright holder, Oracle will change Sun’s current policy and shall not assert or threaten to assert against anyone that a third party vendor’s implementations of storage engines must be released under the GPL because they have implemented the application programming interfaces available as part of MySQL’s Pluggable Storage Engine Architecture.

A commercial license will not be required by Oracle from third party storage engine vendors in order to implement the application programming interfaces available as part of MySQL’s Pluggable Storage Engine Architecture.

Oracle shall reproduce this commitment in contractual commitments to storage vendors who at present have a commercial license with Sun.

This is the biggie, lifting a major cloud from the MySQL storage engine business. It sounds like the third of four options I suggested as to how Oracle could legitimately earn antitrust approval of its MySQL takeover. Sure, Infobright, Kickfire, et al. already had what they saw as adequate safeguards or contingency plans vs. Oracle skullduggery. It’s still big even so.

(Quoted out of order.) The geographic scope of these commitments shall be worldwide and these commitments shall continue until the fifth anniversary of the closing of the transaction.

Not a disaster, but with respect to at least point #2 there should be no time limit whatsoever. I’d like to see the EC require that change as a further Oracle concession.

3. License commitment. Upon termination of their current MySQL OEM Agreement, Oracle shall offer storage vendors who at present have a commercial license with Sun an extension of their Agreement on the same terms and conditions for a term not exceeding December 10, 2014.

Oracle shall reproduce this commitment in contractual commitments to storage vendors who at present have a commercial license with Sun.

Actually, I don’t think this was ever enough of a problem to be a big deal.

4. Commitment to enhance MySQL in the future under the GPL. Oracle shall continue to enhance MySQL and make subsequent versions of MySQL, including Version 6, available under the GPL. Oracle will not release any new, enhanced version of MySQL Enterprise Edition without contemporaneously releasing a new, also enhanced version of MySQL Community Edition licensed under the GPL. Oracle shall continue to make the source code of all versions of MySQL Community Edition publicly available at no charge.

This one is too weasel-worded to matter much.

5. Support not mandatory. Customers will not be required to purchase support services from Oracle as a condition to obtaining a commercial license to MySQL.

Not clear how significant this is given that there are no price assurances about the license cost.

6. Increase spending on MySQL research and development. Oracle commits to make available appropriate funding for the MySQL continued development (GPL version and commercial version). During each of the next three years, Oracle will spend more on research and development (R&D) for the MySQL Global Business Unit than Sun spent in its most recent fiscal year (USD 24 million) preceding the closing of the transaction.

Oracle won’t shut down the MySQL business for at least 3 years. Duh. If there was any chance of Oracle doing so in the first place, it wouldn’t have let MySQL delay the whole merger for this long.

7. MySQL Customer Advisory Board. No later than six months after the anniversary of the closing, Oracle will create and fund a customer advisory board, including in particular end users and embedded customers, to provide guidance and feedback on MySQL development priorities and other issues of importance to MySQL customers.

8. MySQL Storage Engine Vendor Advisory Board. No later than six months after the anniversary of the closing, Oracle will create and fund a storage engine vendor advisory board, to provide guidance and feedback on MySQL development priorities and other issues of importance to MySQL storage engine vendors.

Two small gestures – but I bet the Oracle people tasked to interact with those boards will truly want to do the right thing for MySQL users. Engineers are like that – they want to make things that actually work well.

9. MySQL Reference Manual. Oracle will continue to maintain, update and make available for download at no charge a MySQL Reference Manual similar in quality to that currently made available by Sun.

10. Preserve Customer Choice for Support. Oracle will ensure that end-user and embedded customers paying for MySQL support subscriptions will be able to renew their subscriptions on an annual or multi-year basis, according to the customer’s preference.

Small stuff.

• Like Infobright, Calpont’s InfiniDB is a columnar DBMS consisting of a MySQL front end and a columnar storage engine.
• Community edition InfiniDB runs on a single server.
• One of commercial/enterprise edition InfiniDB’s main claims to fame will be MPP support.
• There’s no announced time frame for commercial edition InfiniDB.
• InfiniDB’s current compression story is dictionary/token only, with decompression occurring before joins are executed. Improvement is a roadmap item.
• Indeed, InfiniDB has many roadmap items, a few of which can be found here. Also, a great overview of InfiniDB’s current state and roadmap can be found in this MySQL Performance Blog thread. (And follow the links there to find performance discussions of other free analytic DBMS.)
• One thing InfiniDB already has that is still a roadmap item for Infobright is the ability to run a query across multiple cores at once.
• One thing free InfiniDB has that Infobright only offers in its Enterprise Edition is ACID-compliant Insert/Update/Delete. (Note: I wish people would stop saying that Infobright Enterprise Edition isn’t ACID-compliant, since that point was cleared up a while ago.)
• InfiniDB has no indexes or materialized views.
• However, InfiniDB’s retrieval is expedited by something called “Extents,” which sounds a lot like Netezza’s zone maps.

Being on vacation, I’ll stop there for now. (If it weren’t for Tropical Storm/ depression Ida, I might not even be posting this much until I get back.)

• Infobright now has >100 paying customers.
• Typical database size is from the low 100s of gigabytes to the low single-digit number of terabytes.
• Agile development is at or approaching two-week release cycles.
• Like Kickfire, Infobright  has a multi-year deal with MySQL that insulates it against many potential Oracle/MySQL shenanigans.
• From an industry perspective, Infobright’s customer base sounds a lot like other vendors’:
  • Data mart outsourcing/online analytics
  • Log files for websites
  • Telecommunications
  • Financial services
  • OEM, especially in the markets cited above
  • “Hey, we’re beginning to see the occasional energy deal”
  • A few random others
• Infobright is seeing some household-name customers, who surely have big-name analytic DBMS products, but who also have a policy that open source is the default choice, and if open source can get the job done then the favorite closed-source choices aren’t used.
• Infobright has the usual open-source community story — lots of involvement and engagement in the forums, but contributions are limited mainly to connectivity, utility scripts, etc. (Maybe some national language translation too; I’m not sure.)

As of yesterday, we were getting 1 1/2 megabytes/hour of “MySQL database has gone away” errors. After Downtown Host declined to discuss that subject with us, Melissa Bradshaw implemented — at least for this blog — a workaround to change the MySQL wait_delay settings ourselves.  Clever idea, and seemed to work for half a day — but now the problems have returned.

Downtown Host isn’t saying much more than “Look at these logs. Your blogs are experiencing a lot of queries and spawning dozens upon dozens of MySQL processes. The main offender is DBMS2.” I don’t know when we’ll get this sorted out. I fly to Europe tomorrow. I have a cough. I’m exhausted. I’m sorry.

I can probably come up with business practices that could make things very hard on Oracle/MySQL competitors … but I haven’t found a compelling antitrust trigger on my first pass over the subject.

Subsequently, there’s been a lot of discussion about whether or not Oracle can use control of MySQL to make life difficult for third-party MySQL storage engine vendors.

Now that the European Commission is delaying the Oracle/Sun deal, explicitly because of Oracle/MySQL antitrust fears.  That is, the European Commission wants to be reassured that an Oracle takeover of MySQL won’t unduly impinge upon the future availability of open source/low cost DBMS alternatives.  This raises that natural question:

What could Oracle do to assure concerned parties that its ownership of MySQL won’t unduly hamper open-source-based DBMS competition?

I think that’s indeed the crucial question. The Oracle/Sun deal has enough momentum at this point that it both should and will be allowed to happen — perhaps with safeguards — rather than banned outright. If  you have concerns about Oracle’s pending acquisition of MySQL, you should speak up and outline what kinds of regulatory safeguards would alleviate the problems you foresee.

More or less obvious possibilities include:

• Divest MySQL. This is obviously an extreme measure, but it surely would work.
• Provide some money and trademark rights to MySQL forkers. If MariaDB and Drizzle were put into strong competitive positions with MySQL today, it’s hard to argue how regulators could object to any future Oracle maneuverings Oracle might envision with the GPLed side of MySQL.
• Offer a standard, attractive, long-term deal to MySQL bundlers. The commercial/non-GPL version of MySQL is a requirement for appliance vendors (surely), OEM vendors (probably), and storage engine vendors (maybe — I disagree, but I’m evidently in the minority).
• Strengthen PostgreSQL. Realistically, that’s not going to be part of any Oracle/MySQL resolution, so I’ll leave it as a subject for another time.

The downside to Continuent Tungsten ’s stripped down architecture is that it doesn’t solve the most extreme performance scale-out problems. Instead, Continuent focuses on the other big benefits of keeping your data in more than one place, namely high availability and data loss prevention (i.e., backup).

Continuent has been around for a number of years, starting out in Finland but now being based in Silicon Valley. For most purposes, however, it’s reasonable to think of Continuent and Tungsten as start-up efforts.

As you might guess from the references to Finland and MySQL, Continuent’s products are open source, or at least have open source versions. I’m still a little fuzzy as to which features are open sourced and which are not. For that matter, I’m still unclear as to Tungsten’s feature list overall …

Fortunately, Kickfire is getting over its secrecy kick. And so, here are some Kickfire technical basics.

• Kickfire is MySQL-based, with all the SQL functionality and lack of functionality that entails.
• The Kickfire/MySQL DBMS is columnar, with the usual benefits in compression and I/O reduction.
• Kickfire is based on FPGAs (Field-Programmable Gate Arrays).
• The Kickfire DBMS is ACID-compliant.
• Kickfire runs only as a single-box appliance.
• While Kickfire earlier estimated that, at least for data sets that compressed well, a Kickfire box could hold 3-10 terabytes of user data, more recent figures I’ve heard from Kickfire have been in the 1-1 /2 terabyte range.  (Edit: Karl Van Der Bergh subsequently wrote in to say that the 1 1/2 TB is raw disk figure, not user data.)

The new information there is that Kickfire relies on an FPGA; Kickfire had long been artfully vague on the subject of FPGA vs. custom silicon. This had the unfortunate effect that people believed Kickfire relied on a proprietary chip, with all the negative implications for future R&D effectiveness that is believed to imply. But in fact Kickfire just relies on standard chips, even if — like Netezza and XtremeData — Kickfire does rely on less programmer-friendly FPGAs to do some of what most rival vendors do on Intel-compatible CPUs.

In terms of how it uses the FPGA, Kickfire is more like XtremeData than like Netezza. That is, large fractions of actual SQL processing seem to be done on the FPGA, not just projections and restrictions. Pipelining is a key concept, in that data is shunted among various “processing engines” without, unless absolutely necessary, being sent back into RAM. If I understood Kickfire founder Raj Cherabuddi correctly:

• There are three kinds of on-FPGA Kickfire “processing engines”.
• Each Kickfire processing engine can do any of about half a dozen different basic things.
• When data finishes at one engine it is sent straight to another engine if at all possible.
• One of the Kickfire optimizer’s main responsibilities is to ensure that this will be possible as often as – well, as often as possible.

Raj says that there are two main reasons data can ever be sent back to memory mid-query. First, the optimizer might sadly fail to find a “networking solution” that allows for perfect pipelining. Second, a query might be so complex that several passes through the pipeline are needed to get it done.

That’s one of Kickfire’s top-two performance strategies. I ran out of time on my last visit before I properly understood the other one, which is something that Kickfire calls “deep indexing,” but which sounds a lot like an inverted list. (Key point: If you have an inverted list already created, joins can be very fast.) When/how exactly that’s used, and what Kickfire does in “hardware” to support it, is a subject for another time.

On the negative side: To get good update/trickle-feed performance, columnar vendors have to do something or other clever. That’s still a future for Kickfire, with the specifics of the roadmap being NDA. I imagine Kickfire also has performance weaknesses in areas where it relies on MySQL for things that MySQL doesn’t happen to be very good at.

*While a few use cases are exceptions, being “better than MySQL” for a DBMS is basically like being “better than Pabst Blue Ribbon” for a beer. Unless price is your top consideration, why are you even making the comparison?

Even worse, the press release, from its subhead and very first sentence, emphasizes the claim “the Groovy SQL Switch’s ability to significantly outperform relational databases.” As CEO Joe Ward quickly agreed by email, that’s not accurate.  As you would expect from the “SQL” in its name, the Groovy SQL Switch is just as relational as the products it’s being contrasted to.  Unfortunately for Joe, who I gather aspires to edit it to say something more sensible, the press release is out already in multiple places.

More favorably, Renee Blodgett has a short, laudatory post about Groovy, with some kind of embedded video.

My client actually signed a deal for EnterpriseDB’s Postgres Plus Advanced Server and GridSQL, but unwound the transaction quickly. (They say EnterpriseDB was very gracious about the reversal.) There seem to have been two main reasons for the flip-flop.  First, it seems that EnterpriseDB’s version of Postgres isn’t up to PostgreSQL’s 8.4 feature set yet, although EnterpriseDB’s timetable for catching up might have tolerable. But GridSQL apparently is further behind yet, with no timetable for up-to-date PostgreSQL compatibility.  That was the dealbreaker.

The current base-case plan is to use generic open source PostgreSQL, with scale-out achieved via hand sharding, Hibernate, or … ??? Experience and thoughts along those lines would be much appreciated.

Another option for OLTP performance and scale-out is of course memory-centric options such as VoltDB or the Groovy SQL Switch.  But this client’s database is terabyte-scale, so hardware costs could be an issue, as of course could be product maturity.

By the way, a large fraction of these updates will be actual changes, as opposed to new records, in case that matters.  I expect that the schema being updated will be very simple — i.e., clearly simpler than in a classic order entry scenario.