Reporter: [Care to comment]?
CAM: SQL Server is an adequate product if you don’t mind being locked into the Microsoft stack. For example, the ColumnStore feature is very partial, given that it can’t be updated; but Oracle doesn’t have columnar storage at all.

Reporter: Is the lock-in overall worse than IBM DB2, Oracle?
CAM: Microsoft locks you into an operating system, so yes.

Reporter: Is this release something larger Oracle or IBM shops could consider as a lower-cost alternative a co-habitation scenario, in the event they’re mulling whether to buy more Oracle or IBM licenses?
CAM: If they have a strong Microsoft-stack investment already, sure. Otherwise, why?

Reporter: [How about] just cost?
CAM: DB2 works just as well to keep Oracle honest as SQL Server does, and without a major operating system commitment. For analytic databases you want an analytic DBMS or appliance anyway.

Best is to have one major vendor of OTLP/general-purpose DBMS, a web DBMS, a DBMS for disposable projects (that may be the same as one of the first two), plus however many different analytic data stores you need to get the job done.

By “web DBMS” I mean MySQL, NewSQL, or NoSQL. Actually, you might need more than one product in that area.

• The most important software on the Oracle Big Data Appliance is a full set of Cloudera Enterprise code. Oracle will do Tier 1 Cloudera/Hadoop support, while Cloudera handles Tiers 2 and 3.
• The key spec ratios are 1 core/4 GB RAM/3 TB raw disk. That’s reasonably in line with Cloudera figures I published in June, 2010.
• This is really Oracle’s multi-structured big data appliance. Oracle’s relational big data appliance is Exadata, which has been out for years and has comparable capacity to Oracle’s new “Big Data Appliance.” (Chris Preimesberger made a similar point.)
• The Oracle Big Data Appliance list price is $450,000 for 18 12-core servers, plus $54,000/year maintenance.
  • That’s around $25,000 per server (and associated storage).
  • That’s also around $2,000/core.
  • That’s also around $500/TB of spinning disk, before compression.
  • None of those per-unit figures sounds ridiculous …
  • … but because of Oracle’s appliance configuration there’s indeed a hefty minimum initial purchase.

Peter Goldmacher argues that, because of size and price point, the Oracle Big Data appliance is targeted for high-end deployments rather than starter/test/development set-ups. To first approximation, that makes sense, in that:

• The Oracle Big Data Appliance is in the petabyte range for data capacity, and …
• … the number of petabyte-scale Hadoop deployments is in the low tens, and …
• … many of those aren’t at Oracle shops anyway.

Surely the Oracle Big Data Appliance isn’t designed for the 4-8 node play-with-Hadoop crowd.

On the the other hand, if you’re at a big, committed Oracle shop, and you want to do your first serious Hadoop deployment, why not go with the Oracle Big Data Appliance? You probably could save money with an alternative approach — but if your employers are committed to Oracle, saving money is surely not their greatest concern. Overpay by a bit; make your management happy with the Oracle logo; get Hadoop on your resume; prosper. That seems like a winning plan all the way around.

• Some version of Cloudera Manager (I’m guessing more or less the best one).*
• Some version of Apache Hadoop (I’m guessing the same distribution that Cloudera prefers to use).*
• Some kind of support.

In other words, it’s a lot like getting Cloudera Enterprise,* plus some hardware, plus some other stuff.

*Edit: About 2 minutes after I posted this, I got email from Cloudera CEO Mike Olson. Yes, the Oracle Big Data Appliance bundles Cloudera Enterprise.

That raises an anyway recurring question: What exactly is Cloudera Manager? When asked, I’ve always tended to mumble something like: Um, it’s management stuff. There’s an overview on the Cloudera Manager product page, but it doesn’t really say much, even if you click on the Data Sheet link. More helpful, I think, is a December post on Cloudera’s busy blog. Technically, the post is about the new features in the Cloudera Manager 3.7 point release, but more generally it helps to explain what Cloudera Manager does, in areas such as (and these bullet points are all direct quotes):

• Automated Hadoop Deployment
• Centralized Management
• Configuration Management
• Service Monitoring
• Log Search
• Events and Alerts
• Configuration versioning and Audit trails
• Activity Monitoring
• Operational Reports

Taken together, those two Cloudera links do a pretty good job of explaining Cloudera Manager, and illustrating why a Hadoop user would want to have either Cloudera Manager or a similar competitive offering.

Edit: The day after I originally made this post, Cloudera put up another post directly explaining what Cloudera Manager is about.

• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

• “Deliver products that really meet customers’ desires and needs.”
• “Successfully convince them that you’re doing so …”
• “… at an attractive overall cost.”

Vendors mentioned here are Oracle, SAP, HP, and IBM. Anybody smaller got left out due to the length of this post. Among the bigger omissions were:

• salesforce.com (multiple subjects).
• SAS HPA.
• The evolution of Hadoop.

A (lingering) issue for SAP and Oracle alike

As I noted in January of this year, integration of business intelligence into operational apps is making very slow progress. Even so, it’s a huge part of the apparent strategy at SAP and Oracle alike, as well it should be. Much of the benefit from automating routine desk work has already happened. The areas ripest for exploitation are the ones where analytics are part of the equation.

Given the lack of tangible progress, why do I think this is a genuine area of Oracle and SAP emphasis? Three reasons of many are:

• Why else did SAP buy Business Objects?
• If they’re not trying to integrate operational apps and analytics, why else does SAP’s emphasis on HANA make sense?
• Without business intelligence in the picture, how does Oracle’s integrated-stack story promise any direct user benefits?*

*As opposed to IT concerns — integration, administration, TCO (Total Cost of Ownership), etc.

After so many years of disappointment, I’m not going to forecast 2012 as a pivotal year for the integration of business intelligence into operational applications. But if one of SAP or Oracle ever does get a significant BI/operational app integration advantage over the other, it could be a major competitive advantage in those application market segments that are still up for grabs. It also is an opportunity for both vendors to gain BI market share in their respective application customer bases.

A more urgent issue for SAP

SAP has put huge amounts of credibility on the line for HANA, the integration of two different and not particularly mature in-memory database technologies. So far, it is difficult to find evidence that HANA is robust enough for widespread adoption. Whether or not SAP can fix that is a huge open question, which could have significant impact on the course of several technology areas: applications, business intelligence, in-memory DBMS, and maybe even hardware.

Based on current information, which is admittedly partial, I’m a short-term pessimist on HANA. Longer-term, I’m on record as saying that traditional databases will eventually wind up in RAM. SAP will surely get that technology right some day, whether or not the way it does so has anything to do with present-day HANA code.

Four more issues for Oracle

Oracle’s ambitions are near-endless, and so also therefore is its list of execution challenges. Four in the analytics area that I find particularly interesting are:

• True hybrid columnar DBMS. I was guessing that Oracle, like Teradata, would announce true hybrid columnar the week of Oracle OpenWorld. I was wrong. But if Oracle can’t bring out true hybrid columnar DBMS functionality relatively soon, Exadata will lose credibility as a competitor to more specialized analytic DBMS.
• Oracle Exalytics. With Exalytics in the mix, Oracle’s technology stack has HANA-like potential. But will Exalytics even ship in 2012? (I think so.) Will it be good for much in the first release? (I’m skeptical.)
• Oracle’s Big Data Appliance. I’m skeptical both about Oracle’s NoSQL product — a favorable InfoWorld review notwithstanding — and Hadoop appliances. But if I’m wrong, and Oracle can successfully embrace/extend the new non-relational paradigms, then it really might regain control over the evolution of data management.
• Oracle’s Endeca acquisition — will Oracle prove me wrong and integrate Endeca effectively into its overall analytic product line? If it does, we might finally see effective text (and eventually speech) navigation of enterprise software. (But as with all Oracle issues cited here, this is something that probably won’t amount to much in 2012 even if it does later go well.)

Three issues for IBM

Like Oracle, IBM is a huge company with many ambitions and hence many execution challenges. The biggest of those is surely: How effective can IBM be at selling outside its existing customer base? I don’t hear as much competitively about IBM DataStage, IBM SPSS or now IBM Netezza as I did when their vendors were independent companies. Even Cognos may not be much of an exception to the rule, although it has its own large customer base outside of IBM’s traditional one. (To lesser extents , the same is of course true of Netezza and numerous other IBM acquisitions.)

Another general issue for IBM is substantively integrating its various product lines, at least to the extent that makes sense. DB2/Netezza integration sounds good, but even that is a matter more of product marketing (the admirable part of that discipline) more than of actual technology. Other integrations (e.g. Cognos/DB2 in various bundles) have tended toward the dubious side.*

*I’m still waiting for IBM to get back to me with examples of how Cognos/DB2 joint tuning amounts to anything. It’s been more than a year, so I’m glad I didn’t hold my breath.

In a somewhat narrower vein, I wonder: Will IBM be able to gain traction for InfoSphere Streams? And if so, when and where will the traction be?

Will HP screw up Vertica?

Vertica has a very attractive product offering. It’s perhaps the most scalable analytic DBMS outside of Teradata, running on the hardware of your reasonable choice.  It’s also the one I recommend most often to clients in the 1-50 terabyte range.

So far HP doesn’t seem to have done much to leadfoot Vertica. (About all I’ve heard from competitors is that Vertica seems to have faded somewhat in the financial services market, and there could be multiple explanations if that is indeed true.) But if HP Vertica does somehow manage to botch things, opportunities will open up for a range of columnar analytic DBMS competitors.

Views of the NoSQL market horse race are reasonably consistent, with perhaps some elements of “Where you stand depends upon where you sit.”

• As James tells it, NoSQL is simply a three-horse race between Couchbase, MongoDB, and Cassandra.
• Max would include HBase on the list.
• Further, Max pointed out that metrics such as job listings suggest MongoDB has the most development activity, and Couchbase/Membase/CouchDB perhaps have less.
• The Cloudera guys remarked on some serious HBase adopters.*
• Everybody I spoke with agreed that Riak had little current market presence, although some Basho guys could surely be found who’d disagree.

*I hope to do a separate post on HBase adoption soon. In connection with that, any info on HBase adoption by Facebook (said to be very heavy), Twitter, et al. would be much appreciated.

The reasons for using NoSQL of course are, in some order, dynamic schemas, scale-out, and open source. I find the scale-out argument somewhat bogus,* but the data model one is very real. Depending on whom you talk with, the most important point about dynamic schemas may actually be that they’re changeable, or it may just be that you don’t have to specify a schema at the time of initial application design. MongoDB gets particular praise as a good platform on which to throw something together quickly, although predictions as to how far the application will then scale may differ depending on whether you’re talking with, say, Max or Todd.

*It’s fair to say that NoSQL systems are more proven in scale-out than most relational DBMS. Even so, I would cringe at any line of reasoning that concluded one should adopt NoSQL because it is more mature than relational alternatives.

Finally, I was perhaps too extreme when I suggested there was no good reason for Oracle to have adopted the major key/minor key approach it took in its NoSQL offering. Todd offered a reason why that approach – which he characterized as similar to Project Voldemort’s – could make sense:

• If you have some kind of global secondary index, it’s hard to maintain that index consistently without what amounts to distributed transactions.
• If you want to avoid the overhead of those, one alternative is a column-group system such as HBase or Cassandra. Those have no indexes at all, except in the sense that a column is its own index.
• Another alternative is to load as much indexing information as you can into the key of a key-value store.

I’d be interested to learn about the Couchbase and MongoDB answers to that challenge.

My favorite relational OLTP scale-out choice these days is the SchoonerSQL/dbShards partnership. Schooner Information Technology (SchoonerSQL) and Code Futures (dbShards) are young, small companies, but I’m not too concerned about that, because the APIs they want you to write to are just MySQL’s. The main scenarios in which I can see them failing are ones in which they are competitively leapfrogged, either by other small competitors – e.g. ScaleBase, Akiban, TokuDB, or ScaleDB — or by Oracle/MySQL itself. While that could suck for my clients Schooner and Code Futures, it would still provide users relying on MySQL scale-out with one or more good product alternatives.

Relying on non-MySQL NewSQL startups, by way of contrast, would leave me somewhat more concerned. (However, if their code is open sourced. you have at least some vendor-failure protection.) And big-vendor scale-out offerings, such as Oracle RAC or DB2 pureScale, may be more complex to deploy and administer than the MySQL and NewSQL alternatives.

The short form of the SchoonerSQL (as Schooner’s product is now called) story goes roughly like this:

• SchoonerSQL replicates data — synchronously if the replication target is local, asynchronously if it is remote.
• Local synchronous replication provides high availability; remote asynchronous replication provides disaster recovery.
• SchoonerSQL’s local synchronous replication also provides read scale-out.
• Schooner has a partnership with Code Futures/dbShards to provide write scale-out via transparent sharding.
• SchoonerSQL has some secret sauce in replication performance. This has the effect of significantly increasing write performance (assuming you were going to replicate anyway), because otherwise you might have to slow down the master server’s write performance so that the slaves can keep up with it.
• Schooner believes it still has some single-server performance advantages as well.

Just to be clear here: Schooner is my client. Code Futures is my client. I introduced them and suggested their partnership. I then introduced them both to a user client, who was sufficiently impressed to seriously evaluate both of them. Even so, some of the Schooner replication story only became clear to me when I visited last Friday.

To flesh that out a bit more:

• Schooner has been making performance tweaks to the MySQL/InnoDB stack for several years. Some of them are still relevant, and can offer 50-100% performance improvements, especially but not only if you use solid-state storage.
• Schooner has found a way to scale up the slave side of master-slave replication, both in the synchronous and asynchronous cases.
  • The core idea of SchoonerSQL parallel (i.e. scale-up) replication is straightforward. The replication log streams in. A chunk is sent off to a CPU core. The next chunk is examined for conflicts with the first chunk. If there are none, it is sent off to a different core to be processed
  • Thus, you can have both local replication/high availability and remote replication/disaster recovery without slowing down writes on the master the way you may have to with other alternatives.
  • Schooner believes this feature alone can make SchoonerSQL 3X faster than MySQL alternatives, at least for writes.

At least in casual conversation, Schooner synthesizes its 1.5-2X single-server figure and up-to-3X clustering figure into a single claim of frequent 2-5X speedup over generic MySQL/InnoDB. But the usual caveats about vendor-supplied performance numbers of course apply.

Finally, some housekeeping:

• At Oracle’s polite request,* Schooner changed its product name to not mention MySQL; hence the moniker SchoonerSQL.
• SchoonerSQL is being launched Monday.
• Schooner has determined that if its version numbers are different from MySQL’s, confusion ensues. So the first ever version under the name SchoonerSQL is SchoonerSQL 5.1, a factoid that Schooner is wisely omitting from the official product launch press release.
• The synchronous part of SchoonerSQL’s replication story dates back to last January’s product release.
• Most of the asynchronous part of SchoonerSQL story is new for Monday.
• dbShards/SchoonerSQL partnership engineering is still underway.

*I mean that non-facetiously. Schooner’s MySQL OEM contract was such that Oracle didn’t have a legal hammer to force the change. Edit: Whoops! There turn out to have been inaccuracies in the original version of this footnote, which I now regret writing. The contract isn’t exactly OEM, and there actually were some trademark-based legal hammers.

Oracle’s core DBMS is not well suited to track interactions (e.g. web clicks), even in cases where it’s the choice for transactions; it’s unnecessarily heavyweight. What’s worse, using the same database to store actions and interactions can lead to serious reliability problems. If a better architecture is to dump the clicks into some NoSQL store, massage the information, and eventually put some derived data into a relational DBMS, then Oracle will naturally try to own each step of the data pipeline.

Dynamic schemas are another area of Oracle weakness, leading in some cases to outright Oracle replacements. However, pure key-value stores go too far to the opposite extreme; you should at least be able to index and retrieve data one field at a time. Based on what I’ve seen of Oracle’s marketing literature, that feature will be missing from the first release of Oracle’s NoSQL.* Until it’s in there, and until it works well, I don’t see why anybody should use Oracle’s NoSQL product.

*Frankly, that choice makes no sense to me on any level. Yet it’s the way Oracle seems to have elected to go — or, if it isn’t, then there’s somebody writing Oracle marketing collateral who’s clearly in the wrong line of work.

In that post, I wrote:

… the Endeca paradigm is really to help you make your way through a structured database, where different portions of the database have different structures. Thus, at various points in your journey, it automagically provides you a list of choices as to where you could go next.

That kind of thing could help Oracle with apps like the wireless telco product catalog deal MongoDB got.

Going back to the Endeca-post quote well, Endeca itself said:

Inside the MDEX Engine there is no overarching schema; each data record carries its own metadata. This enables the rapid combination of a wide range of structured and unstructured content into Latitude’s unified data model. Once inside, the MDEX Engine derives common dimensions and metrics from the available metadata, instantly exposing each for high-performance refinement and analysis in the Discovery Framework. Have a new data source? Simply add it and the MDEX Engine will create new relationships where possible. Changes in source data schema? No problem, adjustments on the fly are easy.

And I pointed out that the MDEX engine was a columnar DBMS.

Meanwhile, Oracle’s own columnar DBMS efforts have been disappointing. Endeca could be an intended answer to that. However, while Oracle’s track record with standalone DBMS acquisitions is admirable (DEC RDB, MySQL, etc.), Oracle’s track record of integrating DBMS acquisitions into the Oracle product itself is not so good. (Express? Essbase? The text product line? None of that has gone particularly well.)

So while I would expect Endeca’s flagship e-commerce shopping engine products to flourish under Oracle’s ownership, I would be cautious about the integration of Endeca’s core technology into the Oracle product line.