• “Hadoop applications” is a catch-all category. Since Derrick offered suitable caveats around the label, I’m fine with what he said.
• Hadoop management software commonly comes in the form of suites. Derrick’s discussion was solid.
• Derrick seems to want to define “Hadoop” as being whatever is in the relevant Apache projects. Cool. He does seem to wind up on both sides of the “MapR and DataStax put Hadoop MapReduce on top of something that isn’t HDFS — so is that Hadoop or isn’t it?” question, but that’s a tough ambiguity to avoid.
• Derrick could have been a little clearer on the subject of Hadoop distributions.

Let’s drill down into that last one. Derrick refers to Hadoop distributions as “products” that:

package a set of Hadoop projects (MapReduce, Hive, Sqoop, Pig, etc.) in a way that in theory makes them integrate more naturally, and to run both smoothly and securely.

While that’s a reasonable recitation of the idea’s benefits, I’d rather say that a “distribution” of open source software comprises:

• Open source software, in selected versions.
• (Possibly) additional code.
• (Likely) documentation.
• (Possibly) legal assurances such as intellectual property indemnification.

In the case of Hadoop:

• The version selection is a relatively big deal. There are a lot of Hadoop sub-projects. There’s been some splitting and forking and recombination. Testing a specific set of point releases for integration and bugs is a non-trivial user benefit.
• The additional code is generally focused on installation or whatever, because the rest is bundled into separately identified management software. Even so, because of the large number of moving parts, this is a good thing to have.
• What’s more, in the case of Cloudera, using a particular distribution (theirs) is a prerequisite to getting the most widely adopted Hadoop management software (also theirs), which in turn is required if you want the industry’s most widely adopted Hadoop support (ditto). Similar things are apt to be true of rival distributions.

• The Forrester Wave’s relative vendor rankings are meaningless, in that the document compares apples, peaches, almonds, and peanuts. Apparently, it covers any vendor that includes a distribution of Apache Hadoop MapReduce into something it offers, and that offered at least two (not necessarily full production) references for same.
• The Forrester Wave for “enterprise Hadoop” contradicts itself on the subject of Hortonworks.
  • The Forrester Wave for “enterprise Hadoop” is correct when it says “Hortonworks … has Hadoop training and professional services offerings that are still embryonic.”
  • Peculiarly, the Forrester Wave for “enterprise Hadoop” also says “Hortonworks offers an impressive Hadoop professional services portfolio”. Hortonworks will likely win one or more nice partnership deals with vendors in adjacent fields, but even so its professional services capabilities are … well, a good word might be “embryonic”.
• Forrester Waves always seem to have weird implicit definitions of “data warehousing”. This one is no exception.
• Forrester gave top marks in “Functionality” to 11 of 13 “enterprise Hadoop” vendors. This seems odd.
• I don’t know why MapR, which doesn’t like HDFS (Hadoop Distributed File System), got top marks in “Subproject integration”.
• Forrester gave top marks in “Storage” to Datameer. It also gave higher marks to MapR than to EMC Greenplum, even though EMC Greenplum’s technology is a superset of MapR’s. Very strange. (Edit: Actually, as per a comment below, there is some uncertainty about the EMC/MapR relationship.)
• Forrester gave higher marks in “Acceleration and optimization” to Hortonworks than to Cloudera and IBM, and higher marks yet to Pentaho. Very odd.
• I’m not sure what Forrester is calling a “Distributed EDW file store connector”, but it sounds like something that Cloudera has provided via partnership to a number of analytic DBMS vendors.
• Forrester’s “Strategy” rankings seem to correlate to a metric of “We’re a large enough vendor to go in N directions at once”, for various values of N.
• Forrester is correct to rank Cloudera’s “Adoption” as being stronger than EMC/Greenplum’s or MapR’s. But Hortonworks’ strong mark for “Adoption” baffles me.

• The most important software on the Oracle Big Data Appliance is a full set of Cloudera Enterprise code. Oracle will do Tier 1 Cloudera/Hadoop support, while Cloudera handles Tiers 2 and 3.
• The key spec ratios are 1 core/4 GB RAM/3 TB raw disk. That’s reasonably in line with Cloudera figures I published in June, 2010.
• This is really Oracle’s multi-structured big data appliance. Oracle’s relational big data appliance is Exadata, which has been out for years and has comparable capacity to Oracle’s new “Big Data Appliance.” (Chris Preimesberger made a similar point.)
• The Oracle Big Data Appliance list price is $450,000 for 18 12-core servers, plus $54,000/year maintenance.
  • That’s around $25,000 per server (and associated storage).
  • That’s also around $2,000/core.
  • That’s also around $500/TB of spinning disk, before compression.
  • None of those per-unit figures sounds ridiculous …
  • … but because of Oracle’s appliance configuration there’s indeed a hefty minimum initial purchase.

Peter Goldmacher argues that, because of size and price point, the Oracle Big Data appliance is targeted for high-end deployments rather than starter/test/development set-ups. To first approximation, that makes sense, in that:

• The Oracle Big Data Appliance is in the petabyte range for data capacity, and …
• … the number of petabyte-scale Hadoop deployments is in the low tens, and …
• … many of those aren’t at Oracle shops anyway.

Surely the Oracle Big Data Appliance isn’t designed for the 4-8 node play-with-Hadoop crowd.

On the the other hand, if you’re at a big, committed Oracle shop, and you want to do your first serious Hadoop deployment, why not go with the Oracle Big Data Appliance? You probably could save money with an alternative approach — but if your employers are committed to Oracle, saving money is surely not their greatest concern. Overpay by a bit; make your management happy with the Oracle logo; get Hadoop on your resume; prosper. That seems like a winning plan all the way around.

• Some version of Cloudera Manager (I’m guessing more or less the best one).*
• Some version of Apache Hadoop (I’m guessing the same distribution that Cloudera prefers to use).*
• Some kind of support.

In other words, it’s a lot like getting Cloudera Enterprise,* plus some hardware, plus some other stuff.

*Edit: About 2 minutes after I posted this, I got email from Cloudera CEO Mike Olson. Yes, the Oracle Big Data Appliance bundles Cloudera Enterprise.

That raises an anyway recurring question: What exactly is Cloudera Manager? When asked, I’ve always tended to mumble something like: Um, it’s management stuff. There’s an overview on the Cloudera Manager product page, but it doesn’t really say much, even if you click on the Data Sheet link. More helpful, I think, is a December post on Cloudera’s busy blog. Technically, the post is about the new features in the Cloudera Manager 3.7 point release, but more generally it helps to explain what Cloudera Manager does, in areas such as (and these bullet points are all direct quotes):

• Automated Hadoop Deployment
• Centralized Management
• Configuration Management
• Service Monitoring
• Log Search
• Events and Alerts
• Configuration versioning and Audit trails
• Activity Monitoring
• Operational Reports

Taken together, those two Cloudera links do a pretty good job of explaining Cloudera Manager, and illustrating why a Hadoop user would want to have either Cloudera Manager or a similar competitive offering.

Edit: The day after I originally made this post, Cloudera put up another post directly explaining what Cloudera Manager is about.

Views of the NoSQL market horse race are reasonably consistent, with perhaps some elements of “Where you stand depends upon where you sit.”

• As James tells it, NoSQL is simply a three-horse race between Couchbase, MongoDB, and Cassandra.
• Max would include HBase on the list.
• Further, Max pointed out that metrics such as job listings suggest MongoDB has the most development activity, and Couchbase/Membase/CouchDB perhaps have less.
• The Cloudera guys remarked on some serious HBase adopters.*
• Everybody I spoke with agreed that Riak had little current market presence, although some Basho guys could surely be found who’d disagree.

*I hope to do a separate post on HBase adoption soon. In connection with that, any info on HBase adoption by Facebook (said to be very heavy), Twitter, et al. would be much appreciated.

The reasons for using NoSQL of course are, in some order, dynamic schemas, scale-out, and open source. I find the scale-out argument somewhat bogus,* but the data model one is very real. Depending on whom you talk with, the most important point about dynamic schemas may actually be that they’re changeable, or it may just be that you don’t have to specify a schema at the time of initial application design. MongoDB gets particular praise as a good platform on which to throw something together quickly, although predictions as to how far the application will then scale may differ depending on whether you’re talking with, say, Max or Todd.

*It’s fair to say that NoSQL systems are more proven in scale-out than most relational DBMS. Even so, I would cringe at any line of reasoning that concluded one should adopt NoSQL because it is more mature than relational alternatives.

Finally, I was perhaps too extreme when I suggested there was no good reason for Oracle to have adopted the major key/minor key approach it took in its NoSQL offering. Todd offered a reason why that approach – which he characterized as similar to Project Voldemort’s – could make sense:

• If you have some kind of global secondary index, it’s hard to maintain that index consistently without what amounts to distributed transactions.
• If you want to avoid the overhead of those, one alternative is a column-group system such as HBase or Cassandra. Those have no indexes at all, except in the sense that a column is its own index.
• Another alternative is to load as much indexing information as you can into the key of a key-value store.

I’d be interested to learn about the Couchbase and MongoDB answers to that challenge.

The other big part of Hortonworks’ story is the claim that it holds the axe in Apache Hadoop development.

and

… just how dominant Hortonworks really is in core Hadoop development is a bit unclear. Meanwhile, Cloudera people seem to be leading a number of Hadoop companion or sub-projects, including the first two I can think of that relate to Hadoop integration or connectivity, namely Sqoop and Flume. So I’m not persuaded that the “we know this stuff better” part of the Hortonworks partnering story really holds up.

Now Mike Olson — CEO of my client Cloudera — has posted his analysis of the matter, in response to an earlier Hortonworks post asserting its claims. In essence, Mike argues:

• It’s ridiculous to say any one company, e.g. Hortonworks, has a controlling position in Hadoop development.
• Such diversity is a Very Good Thing.
• Cloudera folks now contribute and always have contributed to Hadoop at a higher rate than Hortonworks folks.
• If you consider just core Hadoop projects — the most favorable way of counting from a Hadoop standpoint — Hortonworks has a lead, but not all that big of one.

I think Hortonworks likes to make the argument “But our contributions, on average, are more important than Cloudera’s contributions.” That claim perhaps aside, Cloudera’s argument looks persuasive.

Anyhow, the main bases for deciding whose enterprise support for Hadoop to buy — Cloudera’s or Hortonworks’ — are probably:

• Who is even offering it? Hortonworks, last I checked, wasn’t yet — Yahoo perhaps excepted — although it’s a near-term roadmap item for them to start doing so.
• Whose is better? Even when Hortonworks does offer enterprise support, it will lack experience at the support process. (To some extent, that could be worked around by providing money-losingly inefficient support at first.)
• Who bundles more useful proprietary software with their support? Unless you think the code in Cloudera Enterprise is 100% worthless, Cloudera wins that one.
• Price. I have no idea how that one will shake out.

The total number of enterprises in the world paying subscription and license fees that they would regard as being for “Hadoop or something Hadoop-related” probably is not much over 100 right now, but I’d expect to see pretty rapid growth. Beyond that, let’s divide customers into three groups:

• Internet businesses.
• Traditional enterprises ‘ internet operations.
• Traditional enterprises’ other operations.

Hadoop vendors, in different mixes, claim to be doing well in all three segments. Even so, almost all use cases involve some kind of machine-generated data, with one exception being a credit card vendor crunching a large database of transaction details. Multiple kinds of machine-generated data come into play — web/network/mobile device logs, financial trade data, scientific/experimental data, and more. In particular, pharmaceutical research got some mentions, which makes sense, in that it’s one area of scientific research that actually enjoys fat for-profit research budgets.

On the partnering side, I heard things about a Hortonworks conference call that do not seem to have been contradicted by my visit to Hortonworks. Namely, Hortonworks promised prospective partners, such as analytic DBMS vendors, hardware vendors, or large system integrators, that it wouldn’t compete with them, in that Hortonworks pledges not to introduce its own products for at least two years. This is presumably targeted most directly at Cloudera, which has lots of partners, but also some proprietary code of its own. MapR, I’d think, would be the #2 target, but that’s just speculation.

The other big part of Hortonworks’ story is the claim that it holds the axe in Apache Hadoop development. Nobody doubts that a large fraction of the work on Hadoop’s core projects was done by Yahoo employees. Many of those indeed moved to Hortonworks; others left Yahoo earlier; Hadoop creator Doug Cutting is actually at Cloudera. So just how dominant Hortonworks really is in core Hadoop development is a bit unclear. Meanwhile, Cloudera people seem to be leading a number of Hadoop companion or sub-projects, including the first two I can think of that relate to Hadoop integration or connectivity, namely Sqoop and Flume. So I’m not persuaded that the “we know this stuff better” part of the Hortonworks partnering story really holds up.

What I am persuaded of is that the Hadoop platform competition is a good thing. Whichever vendors and projects win will be healthier from having had to outcompete worthy alternatives.

After Michael Stack of StumbleUpon beat me up for a while,* Omer Trajman of Cloudera was kind enough to walk me through HBase usage. He is informed largely by 18 Cloudera customers using, plus a handful of other well-known HBase users such as Facebook, StumbleUpon, and Yahoo. Of the 18 Cloudera customers Omer was thinking of, 15 are in HBase production, one is in HBase “early production”, one is still doing R&D in the area of HBase, and one is a classified government customer not providing such details.

*Just kidding — he was actually extremely gentle.

In the use cases that Omer offered, what’s stored in HBase is almost always records of web or network activity. Specific examples included clickstream information (at 5 different ad companies), crash reports (at Mozilla), and messages (at Facebook). Sometimes the data gets into Hadoop twice — once excerpted via HBase and once as part of a full log — and may even live in two different Hadoop clusters.

What’s served out from HBase in Omer’s examples is usually derived data, such as a user profile, an ad selection, a text index, etc. That makes sense, not least because if you’re going to keep enhancing your data, schema-free programming — which HBase offers — looks ever more appealing. Omer further said that there are a growing number of cases in which HBase is being used to serve up reference data for batch MapReduce jobs, but he didn’t have specifics. A counterexample to the derived data emphasis would be, if I understood correctly, a case where HBase manages shopping carts.

I haven’t put much effort into unearthing open source or other third-party HBase-based projects, but two examples are Open  TSDB  (Time Series DataBase) and Lily CMS (Content Management Systems). (Edit: But see the comment about Lily below.)

Omer is perhaps my top go-to guy on database and cluster sizes, so of course I asked him for HBase metrics as well. He responded (approximately) that Cloudera HBase customer installations average 20-30 nodes, but that half a dozen are in the 100-200 node range.

Finally, there’s the matter of latency. As a general rule, the HBase users Omer sees are using HBase with at least several minutes latency. (Again , that shopping cart case would seem to be a counterexample.) So, for example, the data recorded when you click on a page isn’t immediately applied toward tweaking your profile to determine which ad you’ll see next — but it might come into play after you spend a few minutes reading the page you’re on. Naturally, Omer knows of efforts to use HBase with lower latency yet, and I won’t be surprised if already-working examples of low-latency HBase show up in the comment thread to this post.

• Cloudera’s proprietary code is focused on management, set-up, etc.
• The “Phase 1″ plans Hortonworks shared with me for Apache Hadoop are focused on industrial-strengthness, as are significant parts of “Phase 2″.*
• MapR tells a performance story versus generic Apache Hadoop HDFS and MapReduce. (One aspect of same is just C++ vs. Java.)
• So does Hadapt, but mainly vs. Hive.
• Cloudera also tells me there’s a potential 4-5X performance improvement in Hive coming down the pike from what amounts to an optimizer rewrite.

(Zettaset belongs in the discussion too, but made an unfortunate choice of embargo date.)

*Hortonworks, a new Hadoop company spun out of Yahoo, graciously permitted me to post a slide deck outlining an Apache Hadoop roadmap. Phase 1 refers to stuff that is underway more or less now. Phase 2 is scheduled for alpha in October, 2011, with production availability not too late in 2012.

You’ve probably heard some single point of failure fuss. Hadoop NameNodes can crash, which wouldn’t cause data loss, but would shut down the cluster for a little while. It’s hard to come up with real-life stories in which this has been a problem; still, it’s something that should be fixed, and everybody (including the Apache Hadoop folks, as part of Phase 2) has a favored solution. A more serious problem is that Hadoop is currently bad for small updates, because:

• Hadoop’s fundamental paradigm assumes batch processing.
• Both major workarounds to allow small updates are broken:
  • HBase is seriously buggy, to the point that it sometimes loses data.
  • Storing each update in a separate file runs afoul of a practical limit of 70-100 million files.

File-count limits also get blamed for a second problem, in that there may not be enough intermediate files allowed for your Reduce steps, necessitating awkward and perhaps poorly-performing MapReduce workarounds. Anyhow, the Phase 2 Apache Hadoop roadmap features a serious HBase rewrite. I’m less clear as to where things stand with respect to file-count limits.

Edits: As per the comments below, I should perhaps have referred to HBase’s HDFS underpinnings rather than HBase itself. Anyhow, some details are in the slides. Please also see my follow-up post on how well HBase is indeed doing.

The other big area for Hadoop improvement is modularity, pluggability, and coexistence, on both the storage and application execution tiers. For example:

• Greenplum/MapR and Hadapt both think you should have HDFS file management and relational DBMS coexisting on the same storage nodes. (I agree.)
• Part of what Hortonworks calls “Phase 2″ sets out to ensure that Hadoop can properly manage temp space and so on next to HDFS.
• Perhaps HBase won’t always assume HDFS.
• DataStax thinks you should blend HDFS and Cassandra.

Meanwhile, Pig and Hive need to come closer together. Often you want to stream data into Hadoop. The argument that MPI trumps MapReduce does, in certain use cases, make sense. Apache Hadoop “Phase 2″ and beyond are charted to accommodate some of those possibilities too.

• Packaging up a Cloudera distribution of Apache Hadoop. This distribution doesn’t have proprietary code; it’s just packaged by Cloudera from Apache projects (with a decent minority of the code happening to have been contributed by Cloudera engineers).
• Paid subscription support for Apache Hadoop and, in connection with that …
• …  proprietary software that all support customers automatically get. There are two points to this proprietary software:
  • It adds value for the customer.
  • It makes Cloudera’s support job easier.
• Professional services around Hadoop.
• Training and conferences around Hadoop, which probably don’t generate all that much money, but are great marketing in terms of visibility, thought leadership, and lead generation.

Hortonworks spun out of Yahoo last week, with parts of the Cloudera business model, namely Hadoop support, training, and I guess conferences. Hortonworks emphatically rules out professional services, and says that it will contribute all code back to Apache Hadoop. Hortonworks does grudgingly admit that it might get into the proprietary software business at some point — but evidently hopes that day will never actually come.

Hortonworks’ two main initial marketing messages — and there’s some synergy between these — boil down to:

• Open source purism
• “We have most of the Hadoop developers, so we’re better”*

Frankly, the open source purism part sounds like doubletalk to me, in that Hortonworks has trouble articulating what supposedly-less-pure Cloudera does wrong that Hortonworks will do better. However, I’ve been hearing for a long time that Yahoo’s MapReduce developers feel very strongly about open source, so perhaps this is in part an emotional issue for them. More substantively, it fits well with the pro-Hortonworks story I’ve outlined below.

*”We have most of the Hadoop developers” seems fairly defensible, give or take dueling definitions of “committer,” “core developer,” “patch” or for that matter “Hadoop.”

The other branch of the Hortonworks marketing message can be lampooned as “We’re the right folks to identify your bugs, since we’re probably the ones who put them there in the first place.” More darkly, that pitch could be “If you want the bugs fixed that bother you, we’re the ones who have control over whether or not that happens.” Well, maybe. But I also see Cloudera having a couple years experience supporting Hadoop, as well as shipping some code that perhaps makes Hadoop more supportable.

That’s the skeptical view. A more favorable view of Hortonworks’ prospects would go something like this:

• One version of Apache Hadoop is plenty.
• Cloudera (and arguably other Hadoop platform software vendors) sell capabilities that will soon be eclipsed by core Apache Hadoop. Folks should just please wait.
• Now that Hortonworks is an independent company focused on the task, it will speedily solve the packaging problems that have made Cloudera’s Hadoop distribution (perceived to be) necessary.
• Yahoo and IBM both back Hortonworks’ approach. That’s got to count for something.
• Apache Hadoop will be quickly enhanced, and Hortonworks will be driving the enhancements. Hortonworks simply is the top Hadoop authority.

We’ll see. Cloudera’s been around for a couple years, has smart people, and by definition has no technical inferiority to Hortonworks (since it has access to all Hortonworks’ code). What’s more, it will be a long time before Hadoop technology is so mature that there’s nothing left to do; add-on software should long prove to be useful. As for “We’re purer about open source than the other guys” — well, I’m dubious that that will turn out to be a great marketing message.

And so I think Cloudera is the early favorite in the competition. But perhaps Hadoop users will be able to play Cloudera and Hortonworks off  against each other in price negotiations. Perhaps, notwithstanding my skepticism about Hadoop appliances, some hardware vendors will play them against each other for appliance partnerships.

Meanwhile, whatever else happens, I’m pretty psyched about some enhancements the Hortonworks folks plan to lead for Hadoop.

Related links

• A Hortonworks/Apache Hadoop slide deck Hortonworks graciously allowed me to post
• Cloudera’s post about it’s recent 3.5 release of Cloudera Enterprise
• Pros and cons of professional services efforts at young software companies