• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

• NameNode evolution, and the related issue of file-count limitations.
• JobTracker evolution.

Arun previously addressed these issues and more in a June slide deck.

*Arun has been working on Hadoop full time for over 6 years, and leading the Yahoo/Hortonworks MapReduce team for at least the last 2.

For both NameNode and JobTracker, Arun’s take was along the lines of:

• It’s worked well so far, but it had to be enhanced.
• What we’re taking care of in the current development cycle will make things better.
• Future enhancements after that are of course likely.
• What’s more, we’re doing some refactoring* to improve pluggability, so that alternatives to the mainline Apache Hadoop version of any specific module can easily be swapped in.

*My word, not Arun’s.

In the case of both NameNode and JobTracker, the problem starts with an essential component of the Hadoop system being on a single server. That single server can be mirrored for high availability, but it’s a bottleneck. The current fix for NameNode is to federate it across multiple servers, each owning a portion of the namespace. A variety of further possibilities come to mind:

• Allow persistent storage (I’d urge solid-state memory be assumed in the design), in some sort of a purpose-built system. (Right now NameNode keeps all the metadata it manages in RAM, which is why it has such capacity limitations.)
• Use HBase, as Zettaset does.
• Build on some other open-source DBMS, especially one that was designed for hybrid memory-centric use cases.

The Hadoop guys evidently don’t know which of those approaches will be tried next, in which combination. But if they choose wrong — well, as Arun points out, you’re welcome to implement your own alternative.

Meanwhile, Arun says the current improvement will take Hadoop’s capacity up to half a billion files or so,* or 60-100 petabytes of “storage” (presumably before compression, replication, and so on are taken into account). I didn’t ask Arun to walk me through the arithmetic of that, but I did ask why there were so many files in the first place. I’ve heard in the past of what amounted to “a new file for every update” kinds of scenarios, but that’s not the example he gave. Rather, he spoke of such slicing and dicing of Yahoo advertising data that there are huge numbers of calculated quasi-cubes. For example, advertiser-specific aggregates are run on 5-minute slices of web event data, and stored for 15-minute slices; the raw data is of course kept as well.

*Depending on who you ask, the current figure seems to be in the 70-100 million range, special cases perhaps aside.

Arun also addressed the debate over the fact that MapReduce uses the node-specific file system to create temporary files, rather than HDFS. One reason for this is surely the file-count limitation, but another is performance. Yes, this choice means you lose intermediate results if there’s a node failure; but Hadoop is designed to deal with nodes that fail. And by the way — even if you did use HDFS for intermediate result sets, you might well be setting its replication factor to 1.

As for JobTracker — that’s getting split into at least two pieces, to separate its two main functions:

• Global Resource Manager, which will manage the allocation of resources to various applications, which is apparently a fairly lightweight task.
• Application Master, which will manage the scheduling across nodes of individual applications, which is apparently the part that consumes lots of resources.

Apparently JobTracker has been a key barrier to scalability, being limited to 40-50,000 tasks per cluster, which is a lot like 10-15 tasks/node multiplied by the limit-to-date of 4,000 or so nodes/cluster. The new architecture, in which multiple Application Master compete for resources dispenses by a single Global Resource Manager, is supposed to allow at least 20-50 tasks/node, and a greater number of nodes as well. This capability was checked into the main Hadoop code line just this week.

Global Resource Manager also runs on a single node, high availability mirroring aside. Discussion of opportunities for further scaling was much like our discussion of scaling NameNode — there’s one option that could work, and another option that could also work, and in the mean time one can substitute one’s own version if one likes. Indeed, different Hadoop users already plug different schedulers into JobTracker today.

Another benefit of refactoring JobTracker is that non-MapReduce processing frameworks can be included in Hadoop. The one that gets mentioned repeatedly is MPI (Message Passing Interface). Like the SAS folks, the Hadoop folks seem to think that not all important algorithms can be parallelize via MapReduce, and that MPI is a great place to look for alternative strategies.

I’m still confused about issues such as:

• What resources the Global Resource Manager manages (CPU? RAM? I/O?).
• Whether the Global Resource Manager prioritizes work in the flexible way that, say, a good analytic DBMS workload manager does (I’m guessing it doesn’t, based on the discussion of alternative scheduling modules).
• What, if anything, still needs to be done to actually make MPI work within Hadoop.

But in any case, this all seems to be an important step in Hadoop’s evolution toward being (more of) a flexible, industrial-strength application execution environment.

• Packaging up a Cloudera distribution of Apache Hadoop. This distribution doesn’t have proprietary code; it’s just packaged by Cloudera from Apache projects (with a decent minority of the code happening to have been contributed by Cloudera engineers).
• Paid subscription support for Apache Hadoop and, in connection with that …
• …  proprietary software that all support customers automatically get. There are two points to this proprietary software:
  • It adds value for the customer.
  • It makes Cloudera’s support job easier.
• Professional services around Hadoop.
• Training and conferences around Hadoop, which probably don’t generate all that much money, but are great marketing in terms of visibility, thought leadership, and lead generation.

Hortonworks spun out of Yahoo last week, with parts of the Cloudera business model, namely Hadoop support, training, and I guess conferences. Hortonworks emphatically rules out professional services, and says that it will contribute all code back to Apache Hadoop. Hortonworks does grudgingly admit that it might get into the proprietary software business at some point — but evidently hopes that day will never actually come.

Hortonworks’ two main initial marketing messages — and there’s some synergy between these — boil down to:

• Open source purism
• “We have most of the Hadoop developers, so we’re better”*

Frankly, the open source purism part sounds like doubletalk to me, in that Hortonworks has trouble articulating what supposedly-less-pure Cloudera does wrong that Hortonworks will do better. However, I’ve been hearing for a long time that Yahoo’s MapReduce developers feel very strongly about open source, so perhaps this is in part an emotional issue for them. More substantively, it fits well with the pro-Hortonworks story I’ve outlined below.

*”We have most of the Hadoop developers” seems fairly defensible, give or take dueling definitions of “committer,” “core developer,” “patch” or for that matter “Hadoop.”

The other branch of the Hortonworks marketing message can be lampooned as “We’re the right folks to identify your bugs, since we’re probably the ones who put them there in the first place.” More darkly, that pitch could be “If you want the bugs fixed that bother you, we’re the ones who have control over whether or not that happens.” Well, maybe. But I also see Cloudera having a couple years experience supporting Hadoop, as well as shipping some code that perhaps makes Hadoop more supportable.

That’s the skeptical view. A more favorable view of Hortonworks’ prospects would go something like this:

• One version of Apache Hadoop is plenty.
• Cloudera (and arguably other Hadoop platform software vendors) sell capabilities that will soon be eclipsed by core Apache Hadoop. Folks should just please wait.
• Now that Hortonworks is an independent company focused on the task, it will speedily solve the packaging problems that have made Cloudera’s Hadoop distribution (perceived to be) necessary.
• Yahoo and IBM both back Hortonworks’ approach. That’s got to count for something.
• Apache Hadoop will be quickly enhanced, and Hortonworks will be driving the enhancements. Hortonworks simply is the top Hadoop authority.

We’ll see. Cloudera’s been around for a couple years, has smart people, and by definition has no technical inferiority to Hortonworks (since it has access to all Hortonworks’ code). What’s more, it will be a long time before Hadoop technology is so mature that there’s nothing left to do; add-on software should long prove to be useful. As for “We’re purer about open source than the other guys” — well, I’m dubious that that will turn out to be a great marketing message.

And so I think Cloudera is the early favorite in the competition. But perhaps Hadoop users will be able to play Cloudera and Hortonworks off  against each other in price negotiations. Perhaps, notwithstanding my skepticism about Hadoop appliances, some hardware vendors will play them against each other for appliance partnerships.

Meanwhile, whatever else happens, I’m pretty psyched about some enhancements the Hortonworks folks plan to lead for Hadoop.

Related links

• A Hortonworks/Apache Hadoop slide deck Hortonworks graciously allowed me to post
• Cloudera’s post about it’s recent 3.5 release of Cloudera Enterprise
• Pros and cons of professional services efforts at young software companies

Cloudera can identify 22 CDH (Cloudera Distribution [of] Hadoop) clusters holding one petabyte or more of user data each, at 16 different organizations. This does not count Facebook or Yahoo, who are huge Hadoop users but not, I gather, running CDH. Meanwhile, Eric Baldeschwieler of Hortonworks tells me that Yahoo’s latest stated figures are:

• 42,000 Hadoop nodes …
• … holding 180-200 petabytes of data.

That works out near the low end of the range I came up with for Yahoo’s newest gear, namely 36-90 TB/node. Yahoo’s biggest clusters are little over 4,000 nodes (a limitation that’s getting worked on), and Yahoo has over 20 clusters in total.

Based on those numbers, it would seem that 10 or more of Yahoo’s Hadoop clusters are probably in the petabyte range. Facebook no doubt has a few petabyte-scale Hadoop clusters as well. So we’re probably over 3 dozen petabyte+ Hadoop clusters, just counting Yahoo, Facebook, and CDH users. There surely are others too, running Apache Hadoop without Cloudera’s help.

We also have some more information about the scale of Hadoop usage, and the markets it is being used in, because Omer Trajman of Cloudera kindly wrote the following — lightly edited as usual — for quotation:

The number of Petabyte+ Hadoop clusters expanded dramatically over the past year, with our recent count reaching 22 in production (in addition to the well-known clusters at Yahoo! and Facebook). Just as our poll back at Hadoop World 2010 showed the average cluster size at just over 60 nodes, today it tops 200. While mean is not the same as median (most clusters are under 30 nodes), there are some beefy ones pulling up that average. Outside of the well-known large clusters at Yahoo and Facebook, we count today 16 organizations running PB+ clusters running CDH across a diverse number of industries including online advertising, retail, government, financial services, online publishing, web analytics and academic research. We expect to see many more in the coming years, as Hadoop gets easier to use and more accessible to a wide variety of enterprise organizations.

Omer went on to add:

The biggest number of PB clusters are in the advertising space. I often tell people that every ad you see on the internet touched at least one Hadoop cluster (or the Google equivalent).

• Hadoop nodes today tend to run on fairly standard boxes.
• Hadoop nodes in the past have tended to run on boxes that were light with respect to RAM.
• The number of spindles per core on Hadoop node boxes is going up even as disks get bigger.

A key input comes from Cloudera, who to my joy delegated the questions to Omer Trajman, who wrote:

Most Hadoop deployments today use systems with dual socket and quad or hex cores (8 or 12 cores total, 16 or 24 hyper-threaded). Storage has increased as well with 6-8 spindles being common and some deployments going to 12 spindles. These are SATA disks with between 1TB and 2TB capacity. The amount of RAM varies depending on the application. 24GB is common as is 36GB – all ECC RAM. HBase clusters may have more RAM so they can cache more data. Some customers put Hadoop on their “standard box” which may not be perfectly balanced (e.g. more RAM, less disk) and needs to be altered slightly to meet the above specs. The new Dell C2100 series and the HP SL170 series are both popular server lines for Hadoop.

For a year ago perspective, see this post: http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/

Bullet points from that year-ago link include:

4 1TB hard disks in a JBOD (Just a Bunch Of Disks) configuration

2 quad core CPUs, running at least 2-2.5GHz

16-24GBs of RAM (24-32GBs if you’re considering HBase)

Gigabit Ethernet

So basically we’re talking in the range of 2-3 GB of RAM per core — and 1 spindle per core, up from perhaps half a spindle per core a year ago.

Meanwhile, a 2009 Yahoo slide deck refers to “500 nodes, 4000 cores, 3TB RAM, 1.5PB disk”; that divides out to 8 cores, 6 GB of RAM, and 3 TB of disk per node, all on “commodity hardware.” By 2010 Yahoo was evidently up to 2 GB of RAM per core.

There are lots of data points on the Apache Hadoop wiki, but many seem a few years old, and I don’t immediately see how to time-stamp them. Overall, they seem consistent with the trends I noted at the top of the post.

One thing I haven’t done is attempted to price any of these systems.

Contributions in the comment thread would be warmly appreciated.

Highlights of our visit included:

• There are dozens of people at Yahoo doing Hadoop development that will wind up getting open sourced. (Full-time or close to it.) In particular, everything Mark’s team does goes to open source.
• Yahoo is moving as much of its analytics to Hadoop as possible. Much of this is being moved away from Oracle and from Yahoo’s own Everest.
• A column store is being put on top of HDFS, based on Yahoo technology. Columns will be striped across nodes. Perhaps that’s why the effort is called Project Zebra.
• Mark believes that in a year Hadoop will be much further along in meeting traditional data warehousing requirements, in areas such as:
  • Metadata
  • SLAs/high availability/other workload management
  • Data retention policies
  • Security/privacy*
• Yahoo views the time-to-market benefits of Hadoop as being more important than TCO.

*I also spoke with a couple of Mark’s Yahoo colleagues, on his introduction, who are being less helpful than he is about clarifying what I am or am not allowed to say for publication. But I will say that I was heartened by the degree of concern they showed for doing the right thing with regard to privacy. I was not as heartened by the concrete ideas — or lack thereof — for making it happen. But frankly, I don’t think it’s a solvable technical problem. Rather, it should be a huge priority on the legal/political front.

We also talked some about Pig, Yahoo’s non-SQL DML (Data Manipulation Language) for Hadoop, which is however getting a SQL interface. And we talked about Pig vs. Hive. But I recently heard a rumor all that is in flux, so I won’t write it up now.

Mark sent along a couple of interesting slide presentations by a colleague. After some back and forth as to whether I could post them, he suggested I post these links to similar material instead.

AC Neilsen, for instance, we deployed a 45-terabyte data [mart], they called it; Adidas, 13 terabytes; Australian Bureau of Statistics, 250 terabytes; and of course, some of our high-end ones that you have probably heard of in the past, AT&T, 250 terabytes; Yahoo!, 700 terabytes — just gives you an idea of the size of the databases that are out there and how they are growing, and that’s driving the need for greater throughput.

I don’t know what’s being counted there, but I wouldn’t be surprised if those were legit user-data figures.

Some other notes:

• The Yahoo database is of course Yahoo’s first-generation data warehouse, which has been largely superseded by an internal system more than 10X that size. (Edit: Actually, Greg Rahn of Oracle says below that it’s a different database.)
• I’m keynoting the Netezza road show this month, and Nielsen is up there on stage touting Netezza. (Edit: Nielsen indeed does the overwhelming majority of its data warehousing on Netezza.)
• I’d be surprised if AT&T’s largest data warehouse were “only” 250 terabytes in size. (Edit: Actually, I am told the database in question is 310 TB of user data and growing. More later, hopefully.)
• Oracle didn’t exactly say that those were Exadata installations.

Updating the metrics in my Cloudera post,

• Facebook has 400 terabytes of disk managed by Hadoop/Hive, with a slightly better than 6:1 overall compression ratio. So the 2 1/2 petabytes figure for user data is reasonable.
• Facebook’s Hadoop/Hive system ingests 15 terabytes of new data per day now, not 10.
• Hadoop/Hive cycle times aren’t as fast as I thought I heard from Jeff. Ad targeting queries are the most frequent, and they’re run hourly. Dashboards are repopulated daily.

Nothing else in my Cloudera post was called out as being wrong.

In a new-to-me metric, Facebook has 610 Hadoop nodes, running in a single cluster, due to be increased to 1000 soon. Facebook thinks this is the second-largest* Hadoop installation, or else close to it. What’s more, Facebook believes it is unusual in spreading all its apps across a single huge cluster, rather than doing different kinds of work on different, smaller sub-clusters.

*Apparently, Yahoo is at 2000 nodes (and headed for 4000), 1000 or so of which are operated as a single cluster for a single app.

Facebook decided in 2007 to move what was then a 15 terabyte big-DBMS-vendor data warehouse to Hadoop — augmented by Hive — rather than to an MPP data warehouse DBMS. Major drivers of the choice included:

• License/maintenance costs. Free is a good price.
• Open source flexibility. Facebook is one of the few users I’ve ever spoken with that actually cares about modifying open source code.
• Ability to run on cheap hardware. Facebook runs real-time MySQL instances on boxes that cost $10K or so, and would expect to pay at least as much for an MPP DBMS node. But Hadoop nodes run on boxes that cost no more than $4K, and sometimes (depending e.g. on whether they have any disk at all) as little as $2K. These are “true” commodity boxes; they don’t even use RAID.
• Ability to scale out to lots of nodes. Few of the new low-cost MPP DBMS vendors have production systems even today of >100 nodes. (Actually, I’m not certain that any except Netezza do, although Kognitio in a prior release of its technology once built a 900ish node production system.)
• Inherently better performance. Correctly or otherwise, the Facebook guys thought that Hadoop had performance advantages over DBMS, due to the lack of overhead associated with transactions and so on.

One option Facebook didn’t seriously consider was sticking with the incumbent, which Facebook folks regarded as “horrible” and a “lost cause.” The daily pipeline took more than 24 hours to process. Although aware that its big-DBMS-vendor warehouse could probably be tuned much better, Facebook didn’t see that as a path to growing its warehouse more than 100-fold.  (But based on my discussion with Cloudera, I gather that vendor’s DBMS is indeed used to run some reporting today.)

Reliability of Facebook’s Hadoop/Hive system seems to be so-so. It’s designed for a few nodes at a time to fail; that’s no biggie. There’s a head node that’s a single-point of failure; while there’s a backup node, I gather failover takes 15 minutes or so, a figure the Facebook guys think they could reduce substantially if they put their minds to it. But users submitting long-running queries don’t seem to mind delays of up to an hour, as long as they don’t have to resubmit their queries. Keeping ETL up is a higher priority than keeping query execution going. Data loss would indeed be intolerable, but at that level Hadoop/Hive seems to be quite trustworthy.

There also are occasional longer partial(?) outages, when an upgrade introduces a bug or something, but those don’t seem to be a major concern.

Facebook’s variability in node hardware raises an obvious question — how does Hadoop deal with heterogeneous hardware among its nodes? Apparently a fair scheduling capability has been built for Hadoop, with Facebook as the first major user and Yahoo apparently moving in that direction as well. As for inputs to the scheduler (or any more primitive workload allocator) — well, that depends on the kind of heterogeneity.

• Disk heterogeneity — a distributed file system reports back about disk.
• CPU heterogeneity — different nodes can be configured to run different numbers of concurrent tasks each.
• RAM heterogeneity — Hadoop does not understand the memory requirements of each task, and does not do a good job of matching tasks to boxes accordingly. But the Hadoop community is working to fix this.

Further notes on Hive

Without Hive, some basic Hadoop data manipulations can be a pain in the butt. A GROUP BY or the equivalent could take >100 lines of Java or Python code, and unless the person writing it knew something about database technologically, it could use some pretty sub-optimal algorithms even then. Enter Hive.

Hive sets out to fix this problem. Originally developed at Facebook (in Java, like Hadoop is), Hive was open-sourced last summer, by which time its SQL interface was in place, and now has 6 main developers. The essence of Hive seems to be:

• An interface that implements a subset of SQL
• Compilation of that SQL into a MapReduce configuration file.
• An execution engine to run same.

The SQL implemented so far seems to, unsurprisingly be, what is most needed to analyze Facebook’s log files. I.e., it’s some basic stuff, plus some timestamp functionality. There also is an extensibility framework, and some ELT functionality.

Known users of Hive include Facebook (definitely in production) and hi5 (apparently in production as well). Also, there’s a Hive code committer from Last.fm.

Other links about huge data warehouses:

• eBay has a 6 1/2 petabyte database running on Greenplum and a 2 1/2 petabyte enterprise data warehouse running on Teradata.
• Wal-Mart, Bank of America, another financial services company, and Dell also have very large Teradata databases.
• Yahoo’s web/network events database, running on proprietary software, sounded about 1/6th the size of eBay’s Greenplum system when it was described about a year ago.
• Fox Interactive Media/MySpace has multi-hundred terabyte databases running on each of Greenplum and Aster Data nCluster.
• TEOCO has 100s of terabytes running on DATAllegro.
• To a probably lesser extent, the same is now also true of Dell.
• Vertica has a couple of unnamed customers with databases in the 200 terabyte range.

However, what’s being measured is probably not the same in all cases. For example, I think the Amazon 70 TB figure is obviously for spinning disk (elsewhere in the presentation it’s stated that Amazon has 71 TB of disk). But the 16 TB British Telecom figure probably is user data — indeed, it’s the same figure Computergram cited for BT user data way back in 2001.

The list is:

• Acxiom 16 TB HP
• Allstate 20 TB Sun (RAC)
• Amazon 70 TB HP (RAC)
• AT&T 60 TB HP
• British Telecom 16 TB HP
• Cellcom 12 TB HP
• Choicepoint 14 TB Sun
• Cingular/AT&T 25 TB HP
• Claria 38 TB Sun
• Colgate-Palm 10 TB IBM
• Experian 14 TB Sun
• France Telecom 36 TB HP
• JPMC 40 TB IBM (RAC)
• KTF 14 TB HP
• Mastercard 40 TB IBM (RAC)
• NASDAQ 35 TB Sun
• NexTel 28 TB HP
• NYSE Euronext 93 TB HP (RAC)
• Reliance Ltd 13 TB Sun
• Starwood 12 TB HP
• Sprint/Nextel 110 TB HP
• TIM (Italy)12 TB HP (RAC)
• Turkcell14 TB Sun (RAC)
• UBS AG 15 TB Sun
• UPS 10 TB HP
• Yahoo! 250 TB Fujitsu

I happen to have been on the phone with Phil Francisco of Netezza tonight, and he confirmed that Netezza has larger installations (user data) than the figures cited above at several of those customers, including Axciom and NYSE Euronext. However, Phil confesses that he might have trouble getting up to 10 users at > 15 TB of data if — which I think would be the fairest comparison — he had to restrict himself to only those who have given Netezza permission to publicize their names.*

*Phil emphatically says Netezza has more than that overall. But the customers one is allowed to name, let alone disclose database sizes for, are only a fraction of the overall total.

Meanwhile, I suspect that Reliance might be what turned into one of Greenplum’s flagship accounts. And despite its ongoing Oracle relationship Yahoo has a much bigger data warehouse based on Postgres technology.

As usual, the preponderance of telecom customers is striking.