• PostgreSQL is good technology.
• MySQL is narrowing the gap, but PostgreSQL is still ahead of MySQL in some ways.  (Database extensibility if nothing else.)
• PostgreSQL has a lot of users. (Many of them in academia and/or Russia.)
• Neither EnterpriseDB (which now calls itself “The enterprise PostgreSQL company”) nor the PostgreSQL community leadership have covered themselves with stewardship glory.
• A significant number of interesting DBMS products can be regarded as PostgreSQL forks (e.g. Greenplum, Aster Data nCluster, Netezza if you squint, and Vertica if you stand on your head*).
• PostgreSQL advancement is not dead. For example, Hadapt beta users are running actual PostgreSQL on many nodes each.
• There’s no assurance that Oracle will be a benevolent MySQL steward forever. (Specifically, Oracle’s “Play nicely with others” antitrust commitments expire in 2014.)

So I think it would be cool if one or the other big company put significant wood behind the PostgreSQL arrow.

*While Vertica was originally released using little or no PostgreSQL code — reports varied — it featured high degrees of PostgreSQL compatibility.

• Neil Raden points out that business intelligence dashboards can be dangerously misleading. His reasoning (sound) is that whatever you measure is apt to be distorted by the fact people know they’re being measured. His solution (implied) is to hire a good-looking consultant like himself to do it right.
• I’ve had my issues with Fred Holahan, who was VP of Marketing when I posted that EnterpriseDB was not to be trusted. (That said, Fred is long gone from EnterpriseDB and my opinion hasn’t changed.) But he’s put up a good series of posts on the basis of the open source “progressive engagement” marketing funnel, including this gem on why you shouldn’t count on monetizing your community/free users.
• Oracle plans to increase its acquisition budget. The figure given is $70 billion over the next 5 years. Edit: But see this funny Register followup.
• Clayton Christensen wrote a phenomenal article on how to live a good life, from a very business-y perspective. (Only in one anecdote was it too religiously-oriented for my tastes.) Takeaways include:
  • Your core goals probably revolve around something other than business success. (E.g., family.) Don’t lose sight of that.
  • To the extent you’re a manager or leader, you may have a huge impact on other people’s lives. Use that power in admirable ways.
  • Teach people how to fish for answers, rather than just giving them answers. They’ll probably come to better conclusions than you would have anyway. (This is a core principle in my own consulting.)
  • Take time to reflect. And by the way, the same techniques you use for strategic analysis in business can be applied to your life as well.
• Mark Suster has a pretty good post expanding on my first Christensen takeaway, highlighting a point too often missing from articles in that genre: It’s not just family; it’s also all the cool things around us.
• I haven’t gone through the Hadoop Summit archives yet, but it looks as if there’s a lot of insight there about current Hadoop application activity.
• If you’re a cat lover and don’t hate simple/traditional music, check out Marc Gunn’s cat filksongs, especially the infectious “What Shall We Do With a Catnipped Kitty?” and “Lord of the Pounce”, both playable from the right sidebar of that page (#7 and #10 respectively). Gunn is also a chief perpetrator of the justly (in)famous Do Virgins Taste Better? cycle of filksongs.
• Former SAP exec Dennis Moore offers a theory as to why SAP cares so much about in-memory DBMS. It’s to integrate business processes, because SAP has no other software layer good at doing same. Interestingly, Dennis originated SAP’s previous attempt at meeting a similar need via its composite applications initiative. However, in Dennis’ view this benefit would only be achieved by a major rewrite of SAP’s applications.

*The comment threads are the most informative parts of those posts.

I’m a little unclear as to the Netezza Migrator details, not least because Netezza folks don’t seem to care too much about Netezza Migrator themselves. That said, the core ideas of Netezza Migrator are: 

• Netezza Migrator is an enhanced (?) version of EnterpriseDB’s Postgres Plus Advanced Server DBMS. (Recall that Postgres Plus is PostgreSQL-based and fairly Oracle-compatible.)
• Netezza Migrator does not run on Netezza appliances, but rather on conventional computers off to the side.
• Netezza Migrator generally farms out queries to Netezza appliances, but can also manage data itself. (That latter part could supposedly come in handy for small tables one might want to execute stored procedures against.)
• Netezza Migrator does a better job of farming out queries (and also inserts/updates/loads) to Netezza appliances than an Oracle DBMS would. The two biggest examples of that are:
  • Oracle will farm out SELECTs, but not JOINs.
  • Oracle won’t invoke Netezza’s parallel/bulk load capabilities.

Nonetheless, OLTP/general-purpose data management startup activity has recently picked up, targeting what I see as some very real opportunities and needs. So as a jumping-off point for further writing, I thought it might be interesting to collect a few observations about the market in one place.  These include:

• Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.
• By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value.
• Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.
• It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.
• The era of silicon-centric relational DBMS is coming.
• The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds.
• Users’ instance on “free” could be a major problem for OLTP DBMS innovation.

I shall explain.

Big-brand OLTP/general-purpose DBMS have more “stickiness” than analytic DBMS.

• OLTP applications are more complex than analytic ones, and hence more tightly wired into particular brands of DBMS. For example, third-party packaged OLTP applications are typically portable among only a few brands of DBMS. But third-party business intelligence tools, and the BI “applications” built in them, are more easily and widely portable.
• Specific technical observations such as “OLTP apps tend to use stored procedures, which are DBMS-specific” or “OLTP apps tend to have lots and lots of tables” serve to underscore the first point.
• An enterprise’s highest-value data is commonly the financial stuff handled by its core OLTP systems, so those are the last things they want to mess around with just to get some cost savings. Security, high availability, and so on are major considerations that can outweigh cost.

By number, most of an enterprise’s OLTP/general-purpose databases are low-volume and low-value. Indeed, “OLTP” is often a misnomer, which is why I tend to go with “general-purpose” or some similarly wishy-washy phrase instead.

• In theory, this is a ripe area for what I’ve called mid-range DBMS.
• The big brand vendors try hard to keep as many of those databases for themselves as they can. Enterprise-wide license pricing helps. Going forward, so will virtualization/consolidation strategies, such as Oracle’s Exadata-centric approach.
• A variety of mid-range DBMS alternatives beyond the big brands have technical merit, at least in some cases and configurations – MySQL, PostgreSQL, Intersystems Cache’, and so on.
• The only such mid-range DBMS alternative with much large enterprise business momentum, however, appears to be MySQL.

“General-purpose” might be a better term than “OLTP” anyway.

• I don’t have a link, but it’s widely agreed that over half of the processing on an “OLTP” enterprise app is commonly reporting and so on.
• “Operational BI” is progressing by fits and starts, but it is progressing.
• Anything customer-facing — web-based, call center, or otherwise — is likely to include a heavy dose of “real-time” analytic optimization.

Most interesting new OLTP/general-purpose data management products are either MySQL-based or NoSQL.

• VoltDB is the main exception that jumps to mind.
• This isn’t true in the analytic DBMS area, where Netezza, Greenplum, Aster, Vertica and others started from PostgreSQL’s code, APIs, or both.

It’s not yet clear whether MySQL will prevail over MySQL forks, or vice-versa, or whether they will co-exist.

• MySQL is a limited product without all the third-party storage engines that are being developed.
• Oracle’s promise of MySQL good behavior has an expiration date.
• None of the MySQL front-end alternatives are remotely mature yet.

The era of silicon-centric relational DBMS is coming.

• I think “silicon” means “solid-state memory” as much as or more than it means “RAM,” but that’s not yet certain.
• What is pretty certain is that, thanks to Moore’s Law, some kind of silicon will increasingly replace disk.
• Oracle’s increasingly Flash-centric story is a challenge to everybody.
• RAM-centric VoltDB will launch fairly soon. (By the way, while VoltDB still has a lot in common with H-Store, they’re not exactly the same thing. And H-Store research is progressing too.)
• RethinkDB is being developed, focused directly on solid-state memory. Based on the sparse information available online, RethinkDB sounds somewhat like a dumbed-down H-Store.
• New disk-based vendors may never optimize their use of disk, instead targeting a solid-state future. (E.g., I think Akiban should and quite well might follow this path.)

The emphasis on scale-out and reducing the cost of joins spans the NoSQL and SQL-based worlds. We hear that from the NoSQL guys all the time. But I also just heard it from Akiban.

Users’ instance on “free” could be a major problem for OLTP DBMS innovation. Vendors of new OLTP data management technologies often feel obligated to open source their products, notwithstanding the historical lack of revenue in the open source OLTP DBMS market. As just one of many examples, Nova Spivack wrote:

I have recently seen some new graph data storage products that may provide the levels of scale and performance needed, but pricing has not been determined yet. In short, storage and retrieval of semantic graph datasets is a big unsolved challenge that is holding back the entire industry. We need federated database systems that can handle hundreds of billions to trillions of triples under high load conditions, in the cloud, on commodity hardware and open source software. Only then will it be affordable to make semantic applications and services at Web-scale.

I hear similar things from other startups, who evidently believe they need and/or are entitled to enjoy sophisticated, high-performance, zero-cost, specialized database management technology.

Greenplum Single-Node Edition:

• Is free of charge, although you can buy support.
• Has no restrictions on use, production or otherwise.
• Has no restrictions on database size.
• Is closed-source.

For those who want free, terabyte-scale data warehousing software, Greenplum Single-Node Edition may be quite appealing, considering that the main available alternatives are:

• General-purpose open-source DBMS, such as PostgreSQL and MySQL (lacking analytic DBMS performance and features)
• Infobright Community Edition (the other best choice – Infobright’s commercial sales success indicates the solidity of Infobright’s technology)
• Rough research-project code and other other questionable open source offerings
• Crippleware from other commercial analytic DBMS vendors (e.g., Teradata)

For example, comparing PostgreSQL-based Greenplum with PostgreSQL itself, Greenplum offers:

• The ability to scale out queries across all cores in your box (and no, pgpool is not a serious alternative)
• Storage alternatives such as columnar (I am told that EnterpriseDB recently stopped funding a project for a PostgreSQL columnar option)

Greenplum would surely also argue that its software is superior to PostgreSQL in parallel load, compression, MapReduce integration, and general fit-and-finish. I imagine that in some (perhaps not all) cases it would be right. PostgreSQL’s main technical advantages over Greenplum would probably lie in the area of datatype extensibility.

The main target users for Greenplum’s Single-Node Edition are obviously individual enterprise power users or very small analytic teams. I.e., it’s people with a data mart need that a central data warehouse isn’t meeting. Potential benefits to Greenplum include:

• Adding value to its Enterprise Data Cloud story
• Seeding the market for future enterprise sales
• Depriving competitors of revenue, perhaps at enterprises too small to ever be paying Greenplum customers

In addition, I see free Greenplum as a charity offering that could be appealing to scientists who face PostgreSQL performance limitations.

Related links

• Greenplum Free Single-Node Edition press release (I’m quoted)
• MySQL Performance blog on MonetDB and Infobright community edition
• PostgreSQL’s restriction to one core per query
• Infobright’s restriction to one core per query

I have a client who wants to build a new application with peak update volume of several million transactions per hour.  (Their base business is data mart outsourcing, but now they’re building update-heavy technology as well. ) They have a small budget.  They’ve been a MySQL shop in the past, but would prefer to contract (not eliminate) their use of MySQL rather than expand it.

My client actually signed a deal for EnterpriseDB’s Postgres Plus Advanced Server and GridSQL, but unwound the transaction quickly. (They say EnterpriseDB was very gracious about the reversal.) There seem to have been two main reasons for the flip-flop.  First, it seems that EnterpriseDB’s version of Postgres isn’t up to PostgreSQL’s 8.4 feature set yet, although EnterpriseDB’s timetable for catching up might have tolerable. But GridSQL apparently is further behind yet, with no timetable for up-to-date PostgreSQL compatibility.  That was the dealbreaker.

The current base-case plan is to use generic open source PostgreSQL, with scale-out achieved via hand sharding, Hibernate, or … ??? Experience and thoughts along those lines would be much appreciated.

Another option for OLTP performance and scale-out is of course memory-centric options such as VoltDB or the Groovy SQL Switch.  But this client’s database is terabyte-scale, so hardware costs could be an issue, as of course could be product maturity.

By the way, a large fraction of these updates will be actual changes, as opposed to new records, in case that matters.  I expect that the schema being updated will be very simple — i.e., clearly simpler than in a classic order entry scenario.

• Bob Zurek, EnterpriseDB’s well-regarded CTO, is gone. (He landed at Infobright, after a stint of independent consulting.)
• Multiple rumors have founder Andy Astor leaving EnterpriseDB, and stepping back to an advisory role. One version has Tuesday, June 16 as Andy’s last day. Update: As of Wednesday, June 17, Andy Astor is no longer listed as being on EnterpriseDB’s management team.
• Fred Holahan, who was briefly VP of Marketing, is not listed on  EnterpriseDB’s management team web page.  And EnterpriseDB announced a new VP of Marketing and Product Management on May 21.
• Other rumors point to turmoil at EnterpriseDB as well.

And by the way, EnterpriseDB, which used to call itself “the Oracle-compatible database company,” recently licensed out what used to be its core differentiating technology.

Now, this isn’t all bad news. EnterpriseDB’s Oracle-compatibility focus had to be changed anyway. And Fred Holahan was the proximate cause for me writing:

my recent dealings with EnterpriseDB underscore the importance of being VERY careful about counting your fingers after you shake hands with that company,

Still, these aren’t exactly indicators of a company executing on a smooth-running plan.

Key points include:

• This really is more like Oracle emulation than it is transparency, a term I carelessly used before.
• IBM’s Oracle emulation effort is focused on two technological goals:
  • Making it easy for an Oracle application to be ported to DB2.
  • Making it easy for an Oracle developer to develop for DB2.
• The initial target market for DB2′s Oracle emulation is ISVs (Independent Software Vendors) much more than it is enterprises. IBM suggested there were a couple hundred early adopters, and those are primarily in the ISV area.

Because of Oracle’s market share, many ISVs focus on Oracle as the underlying database management system for their applications, whether or not they actually resell it along with their own software. IBM proposed three reasons why such ISVs might want to support DB2:

• Oracle is expensive. In particular, IBM suggested it is more flexible on licensing terms for resale than Oracle is. I find that easy to believe.
• Hey, there’s a DB2 market or installed base out there of some size — why not address it?
• Acquisition-fueled expansion in applications makes Oracle a much bigger competitor to many ISVs (all around the world) than it used to be before. That one makes all kinds of sense.

And by the way — if I wanted an Oracle-emulating DBMS, I’d feel a lot happier about doing business with IBM than I would with EnterpriseDB.

IBM feels that DB2′s Oracle compatibility is a strict superset of EnterpriseDB’s, which it presumably has carried over more or less in its entirety. I didn’t press too hard for examples of what Oracle emulation DB2 offers and EnterpriseDB doesn’t, but IBM did say something about support for more programming languages. IBM was clear on one broad area where DB2 does not offer Oracle emulation, which is the specifics of various kinds of datatype support or other specialized data access methods. For example, IBM has its own syntax for querying text, geospatial, or XML data, and has not added support for Oracle’s alternative approaches.

*The looseness in that definition highlights a key reason these technologies don’t sell well — it’s hard to be sure that what you’re buying will do a good job of running your particular apps.

This subject comes to mind for two reasons. One is that IBM seems to have licensed EnterpriseDB’s Oracle transparency layer for DB2. The other is that a natural upgrade path from MySQL to Oracle might be a MySQL transparency layer on top of an Oracle base.

At first blush, the Oracle/MySQL possibility could break the mold. Migrating from one product to another product owned by the same vendor is a lot different than migrating from one vendor’s product to another’s. Users have tremendous familiarity with upgrades where one vendor controls both the start and end points of the transition.

On the other hand, the number of cases where a vendor has bought a DBMS product and then migrating a substantial user base over to another DBMS is approximately zero. The template for reasonably successful DBMS vendor consolidations — such as IBM/Informix or Oracle/RDB — is almost always to maintain and enhance multiple product lines side by side.

As for EnterpriseDB/DB2 — if you have an application running on Oracle, why port it to DB2? Unless IBM gets aggressive on its maintenance licensing terms, that won’t even get you much of a first-glance cost saving. And while it’s annoying to do DBA work for two database brands when one will suffice — if you have those Oracle apps already running, then you also already have the DBA resource to keep them going. No doubt there will be situations where this new offering is useful and welcome, but they’ll probably prove to be rather isolated edge cases.

A couple of years ago, I did make a theoretical argument that DBMS portability should become technically easier and hence more widely adopted. But since then I’ve seen very little practical evidence to back it up.

]]> http://www.dbms2.com/2009/04/22/dbms-transparency-layers-never-seem-to-sell-well/feed/ 11 http://www.dbms2.com/2009/04/20/first-thoughts-on-oracle-acquiring-sun/ http://www.dbms2.com/2009/04/20/first-thoughts-on-oracle-acquiring-sun/#comments Mon, 20 Apr 2009 12:57:04 +0000 Curt Monash http://www.dbms2.com/?p=755

More later.  I have a radio interview in a few minutes on a very different subject.