• Traditional data of the sort that’s stored in relational databases, aka “structured.”
• Everything else, aka “unstructured” or “semi-structured” or “complex.”

There are plenty of problems with these formulations, not the least of which is that the supposedly “unstructured” data is the kind that actually tends to have interesting internal structures. But of the many reasons why these distinctions don’t tend to work very well, I think the most important one is that:

Databases shouldn’t be divided into just two categories. Even as a rough-cut approximation, they should be divided into three, namely:

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays
• Human/Nontabular data — i.e., all other data generated by humans
• Machine-Generated data

Even that trichotomy is grossly oversimplified, for reasons such as:

• These categories overlap.
• There are kinds of data that get into fuzzy border zones.
• Not all data in each category has all the same properties.

But at least as a starting point, I think this basic categorization has some value.

By human-generated data that fits well into relational tables or arrays, what I really mean is: the input from most conventional kinds transactions – purchase/sale, inventory/manufacturing, employment status change, etc. This is the core data managed by OLTP relational DBMS everywhere. It is also the core data in analytic relational or MOLAP databases. The vast majority of what we think or know about “database management” applies primarily to data of this kind, in large part because of two fundamental properties of this information:

• It is meaningful to contemplate this data as being 100% accurate and complete (even if that goal is difficult to achieve in the real world).
• This data is precise – i.e., one can check predicates against it and (give or take regrettable data imperfections) get inarguable yes/no answers.

For most enterprises, this is the most important data they have. It was created as a result of expensive business activities. It deals directly with money, employees, physical goods, and the rest of the things that make an enterprise go. It can be fruitfully analyzed in ever more ways, which is why it should never be thrown out or even entirely relegated to tape, now that data warehouse software, hardware, and storage has become so cheap. (“Disk is the new tape.”) And because of the importance of both preserving and accessing it, it should often be stored in multiple copies – OLTP, data warehouse, data mart, in-memory analytics, near-line quasi-archive, MOLAP cubes (if you must) and so on, plus of course replicas for high throughput and availability.

But humans generate many other kinds of data as well, especially in a form directly suitable for communication – text (in many formats), documents (text or otherwise), pictures, videos, etc. Traditional relational databases are a poor home for this kind of data because:

• This data often deals with opinions or aesthetic judgments – there is little concept of perfect accuracy.
• Similarly, there is little concept of perfect completeness.
• There’s also little concept of perfectly, unarguably accurate query results – different people will have different opinions as to what comprises good results for a search.
• Queries don’t lend themselves to binary answers; rather, documents can have differing degrees of relevancy.

Systems for managing this kind of data are much less advanced than relational database managers. Nobody knows how to get all the information out of a text document, or query all of it if they could, and the story is even worse for non-text examples. The systems that give the best query results aren’t necessarily the same ones that have the best database administration features. Basically, this area is still a mess, and it’s a mess that consumes a huge fraction of all the data storage products sold today.

But give or take questions of storage efficiency and deduplication, if humans created that kind of data, they put a lot of effort into it, so it’s worth keeping. Besides, compliance regulations commonly mandate that we do so – except, perhaps, when they mandate that we throw it away.

Machine-generated data is a whole other can of worms. Paradigmatic examples of what I mean by “machine-generated data” include:

• Computer, network, and other equipment logs
• Satellite and similar telemetry (whether for espionage or science)
• Location data such as RFID chip readings, GPS system output, etc.
• Temperature and other environmental sensor readings
• Sensor readings from factories, pipelines, etc.
• Output from many kinds of medical device, in hospitals and (increasingly) homes alike

Unlike human-generated data, whose growth is constrained by macro factors such as population and total level of economic activity, machine-generated data will continue to grow as fast as Moore’s Law lets it. That fact has two profound consequences:

• It is unrealistic to hope ever to keep most or all machine-generated data, whereas I think that’s exactly what should and will happen with human-generated data
• Before long, most data (by volume) will be machine-generated

And so it is not really an exaggeration to say that machine-generated data is the future of data management.

I’d like to close this long post by immediately pointing out some of the flaws in this simple trichotomy. One obvious gray area lies in hybrid human/machine-generated data, three big examples of which are:

• Web clickstreams
• Call detail records (CDR)
• Stock trades

In all three cases, we are quickly getting to the point where this data is preserved in its entirety (even if the network event data associated with the web logs is reduced before storage). And in each case it fits pretty well into RDBMS, although Hadoop has a role to play as well. So pretending it’s purely human-generated probably isn’t all that misleading.

Another gray area lies in text that gets linguistically processed – i.e. via text-mining tools – with the output placed into a relational database. I don’t immediately see a workaround for that flaw in my labeling scheme.  So let’s just say no taxonomy is perfect.*

*Come to think of it, that’s one of the problems holding back text-mining technology.

And of course some of the NoSQL folks would note that I was oversimplifying when I tied my first category specifically to relational DBMS. So would the folks at Intersystems.

But the biggest oversimplification stems from this:

As Mike Stonebraker* and I argued a couple of years ago, I really think that database management technologies should be divided into 10+ categories.

*Note: The links to Stonebraker’s own posts will be broken until Vertica’s webmaster gets his/her act together. But you can find them under other URLs via web search.)

]]> http://www.dbms2.com/2010/01/17/three-broad-categories-of-data/feed/ 9 http://www.dbms2.com/2009/12/07/data-warehouse-volume-growth/ http://www.dbms2.com/2009/12/07/data-warehouse-volume-growth/#comments Mon, 07 Dec 2009 13:50:47 +0000 Curt Monash http://www.dbms2.com/?p=1278 There are only three ways that the amount of data stored in data warehouses can grow:

• The same kinds of data are stored as before, with more being added over time.
• The same kinds of data are stored as before, but in more detail.
• New kinds of data are stored.

The first of those three ways doesn’t lead to dramatic growth. If a data warehouse goes up from 5 years of data to 6, then its overall size will grow a little over 20%. (How little depends on what the underlying business growth is – i.e., on how many more business events you have next year than you had 3 years ago.) That’s almost certainly going to be well-handled, by whatever technology manages your data warehouse today, given that:

• Chips are still subject to something resembling Moore’s Law.
• Disk capacity is still subject to Kryder’s Law, which is like Moore’s Law but with yet faster growth rates.
• DBMS software gets more performant over time.

So the cost of managing your same-as-before data will go down every year, even as the volume of that data grows.

True, disk rotation speeds have only increased 12.5 times since the Eisenhower Administration. But solid-state drives (SSDs) are getting practical for data warehousing fast, so even that bottleneck eventually will get swept away. And since what we’re discussing is, basically, the first and hence presumably highest-value data to be warehoused, it’s apt to wind up on SSDs before some other kinds of data warrant that treatment. So it’s the two other factors that drive the greatest data warehouse growth.

As costs go down, the wisdom of keeping detailed data goes up. I’d go so far as to say that every piece of data generated by a human being should be preserved and kept online, legal and privacy considerations permitting.* Most forms of capital-, labor-, and/or location-based competitive advantage being commoditized and/or globalized away. But information remains a unique corporate asset. Don’t discard it lightly.

*Unless there’s an explicit law mandating data destruction, legal considerations should permit. The idea “Let’s destroy something of irreplaceable value today, against the possibility we might be brought to judgment tomorrow” is both morally and pragmatically weird. Privacy, however, may be a different matter.

What that means in practice is that “disk is the new tape.” No-apologies performance can be had on data warehouse systems for $20,000/terabyte or less – perhaps even a lot less. Tolerable performance may cost 3-4X less than that. I think a lot of the growth in data warehouse volumes is of exactly this kind.

Ultimately, however, the greatest growth in data warehouse volumes will come from new kinds of data, especially data that is partly or wholly machine-generated. Moore’s Law applied to sensor chips tells us that data creation will grow just as fast as the data storage capacity. And thus we will be throwing away most machine-generated data forever. But what we keep will grow – well, it probably will grow at Moore’s/Kryder’s Law speeds.

That’s not to say new kinds of data are all high-volume/machine-generated. Back in 2005, I wrote two pieces for Computerworld advocating aggressive pursuit of new data sources, and the examples I mentioned were:

• Loyalty cards, especially in gaming
• Location-based analytics
• Extra customer feedback (e.g., opinion surveys)
• Price/offer testing
• Text mining in general
• Medical records

Today I’d add (among others):

• RFID
• The raw output from medical test devices
• Sensors up and down the energy supply chain

But some of those older, low-data-volume ideas still head my list of low-hanging analytic fruit.

One more complication – these buckets I’m outlining are less than precise. For example:

• Telecom CDRs (Call Detail Records) are machine-generated from a seed of human activity. They have long been stored, but now are being kept in much more detail. This is why telecommunications is one of the top markets for data warehouse technology.
• Stock trade data used to be based on human decisions. Now most of it is just machines buying and selling from each other. Either way, increasingly many investment institutions want to keep 100-terabyte-scale databases of complete historical trade detail. And that is why financial services is another huge market for data warehouse technology.
• Not long ago, web and network event logs. didn’t even exist, or were tiny where they did. Now they fill the largest known commercial databases, at firms such as Yahoo, eBay, and Facebook. Even so, more is thrown away than kept, especially on the network event side, which is a multiple of the size of the pure clickstream data.
• We don’t know exactly what all data intelligence agencies collect from telemetry, from monitoring commercial telecommunication traffic, and so on. But they’re surely throwing the vast majority away, even as the small part they keep is petabyte-scale.

But none of that interferes with my main points, which are:

• Databases will continue to grow very quickly.
• One big driver is the increasing detail in which data is kept online.
• An even bigger driver will be the unending ability of machines to generate ever greater streams of at-least-somewhat interesting data.

• Registration for tomorrow’s webinars
• Replay of the first webinar
• My slides from the first webinar

The main subjects of the webinar will be:

• Some review of material from the first webinar (all three presenters)
• Discussion of how MapReduce can help with three kinds of analytics:
  • Pattern matching (Jonathan will give detail)
  • Number-crunching (I’ll cover that, and it will be short)
  • Graph analytics (I haven’t written the slides yet, but my starting point will be some of the relationship analytics ideas we discussed in August)

Arguably, aspects of data transformation fit into each of those three categories, which may help explain why data transformation has been so prominent among the early applications of MapReduce.

As you can see from Aster’s title for the webinar (which they picked while I was on vacation), at least their portion will be focused on customer analytics, e.g. web analytics.

The top two points from Q&A probably were:

• Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
  • Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
  • Getting data into the cloud in the first place is a lot slow.
• The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

• Big Data vs. cloud
• How big is Big Data?
• At the low end of that range, there’s little you can’t do with conventional technology if you have:
  • An unlimited budget for hardware
  • An unlimited budget for software
  • An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

• Hard-core OLTP
  • Focus of DBMS technology for a long-time
  • Big budgets because each transaction has significant value
  • Tough to get users to change technologies
• Lighter-weight OLTP
  • Classic example = web companies
    • Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
    • Reluctant to give money to anybody
      • Open source, etc.
  • Difficulty finding market
    • Product vs. feature
      • Clustering/HA/DR/whatever
      • Ditto cloud enablement
    • True products haven’t found much traction yet

Analytic Big Data use cases

• Kinds of data for analytics
  • More of same != big
  • More detail and/or new kinds
    • Complete data sets
    • Transactions
    • Call details
    • Tick/trade history
    • Web clickstreams
    • Network event logs
    • Other machine-generated data
    • CAM bottom line
      • Anything human-generated should and will be retained in its entirety
      • Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
• Analytic uses of Big Data
  • Analytics is mainly about three things
    • Problem detection
    • Customer relationship improvement
      • (Those overlap when the customer relationship is bad)
    • Financial statements on steroids
  • Main kinds of analytics
    • What BI vendors traditionally sell
      • General reporting and dashboards
      • Ad-hoc query (now driven from those reports and dashboards)
      • Planning (allegedly integrated with BI)
    • Research
      • Ad hoc relational query (worth mentioning twice because it drives so much of the market)
      • Data mining
      • Most web search and web mining
    • Operational/near-real-time
    • Archiving/compliance
  • What gets Big?
    • Mainly research and archiving
    • But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

• Moore’s Law
  • CPUs — All about cores, hence parallelism is key
  • RAM
  • SSDs – hence replace disks
  • Sensors – hence generate lots more data
• Kryder’s Law
  • But rotational speeds up only 12.5X since Eisenhower Administration
  • Hence solid-state memory (or RAM) will soon take over
• In the mean time, I/O bottlenecks have had to be beaten
  • Hence sequential scans
  • Hence index-light architectures
  • Hence columnar
• DBMS “overhead”
  • Raw license and maintenance fees – software increasing fraction of total
  • OLTP vestiges – locking and all that
  • DBAs
    • People costs = huge fraction of total
    • Index-lightness addresses
    • So does appliance
  • Many people don’t really know how to write SQL
• Configuration
  • Appliance/tightly-balanced
    • Netezza
    • Teradata earlier
    • Greenplum/Sun
    • Oracle
    • IBM
    • Microsoft/Madison
  • Commodity/do what you want
    • Vertica
    • Greenplum now
    • Infobright, Aster and others
    • MapReduce-oriented file systems
  • Extreme rigidity is silly
    • Teradata, Oracle have both signaled moving to more modularity
    • Big driver of that = heterogeneous storage
      • Cheap disk
      • Expensive disk
      • Solid-state
      • RAM
    • CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

• Latency requirements
  • High availability and low latency go together
• Query types
  • Simultaneous users for same
• Database size
• Budget

Actual segments right now

• Utter ADW/EDW
• Data mart
  • Size
  • Naturally columnar vs. naturally row-based
• Operational/frontline
• Less dramatic/smaller EDW

• MapReduce is something very new
• MapReduce involves strict adherence to the Map-Reduce programming paradigm
• MapReduce is a single technology

So let’s give it a try.

When Dave DeWitt and Mike Stonebraker leveled their famous blast at MapReduce, many people thought they overstated their case. But one part of their story – one that both Mike and Dave say was most central to their case – was never effectively refuted, namely the claim that these ideas aren’t particularly new. I haven’t actually read enough computer science literature to have an independent opinion on that issue. But I’ll say this – claims from companies such as SenSage, Oracle, or Splunk that “We’ve been doing MapReduce all along” seem pretty credible to me.

True, what those companies were doing things may not have looked exactly like the instant-classic MapReduce programming paradigm. But the same is true of many things almost everybody would agree count as MapReduce. In particular, it is often not the case that you alternate Map and Reduce steps, each of whose outputs is a set of simple <Key, Value> pairs, with data redistributed based on Key at every step.

Here are some examples of what I mean, drawn from my recent MapReduce webinar.

• If you do text indexing in MapReduce, your goal is to wind up with a text index. So at some point you Reduce to a pair <WordName, {all the (DocumentID, offset) pairs for the whole corpus, suitably ordered}>. That’s a heckuva compound “Value”.
• The goal of data mining is usually to estimate a rather small number of parameters based on a large overall data set, often – depending on algorithm – in the form of a single vector. When you do that in MapReduce. you partition data among nodes, calculate something on each node that is structured more or less like your final vector. So when it comes time for the reduce, you just ship all of your vectors – one per node – to a single Reduce node, and do the appropriate math. Redistribution based on Key would be quite pointless.
• When you sessionize clickstream logs in MapReduce, you may have just as many output records as input records. However, they now are reformatted, and might have a SessionID appended. In those cases, Reduce isn’t doing much by the way of reduction.
• And as I happens in some Vertica-Hadoop use cases around mortgage trading, sometimes MapReduce can even make data sets vastly larger.

By no means do I think this is a weakness of the MapReduce programming paradigm. Rather, I think it’s a MapReduce strength. But it’s not quite the way MapReduce has been promoted and explained to the IT public.

Finally: MapReduce, as commonly conceived, spans two different – albeit closely related – technology domains:

• Parallel programming
• Distributed data management

For example, I imagine Greenplum’s and Vertica’s MapReduce/SQL combined syntaxes are very similar to each others. But Vertica’s data management implementation of MapReduce, which relies on Hadoop, is very different from Greenplum’s, which is tied into the Greenplum DBMS. Similary, non-DBMS MapReduce implementations are commonly associated with distributed file systems – notably HDFS (Hadoop Distributed File Systems) or Google’s internal GFS (Google File System). In those systems, the parallel language execution part should be aware of how the distributed file management part works – but perhaps that awareness can be pretty lightweight.

Right now, this is a distinction pretty much without a difference. If you choose an implementation of MapReduce — like pure Hadoop (say in the Cloudera distribution) or Hadoop-Vertica or Aster Data’s SQL/MapReduce – you’re basically picking an entire technology stack. But those stacks are going to do a whole lot of changing and maturing in the near future – and as they do, it’s likely that projects will interact or even combine in all sorts of interesting ways.

Bottom line: There are a lot of different ways to exploit MapReduce-related technology.

Splunk’s core technology indexes text and XML files or streams, especially log files. Technical highlights of that part include:

• Splunk software both reads logs and indexes them. The same code runs both on the nodes that do the indexing and on machines that simply emit logs. However, in the latter case indexing is turned off. Thus, Splunk does not portray its software as “agentless.” However, it asserts that its agent-like software runs without “material” overhead.
• The fundamental thing that Splunk looks at is an increment to a log – i.e., whatever has been added to the log since Splunk last looked at it.
• Splunk tries to figure out what the individual entries are in a section of log it looks at. In particular:
  • Time stamps are a big clue in this “inferencing” process, but they are not the be-all and end-all.
  • Nor are line boundaries, if logs are naturally broken up into lines. (Splunk threw that latter comment in as a shot at SenSage.)
• I get the impression that most Splunk entity extraction is done at search time, not at indexing time. Splunk says that, if a <name, value> pair is clearly marked, its software does a good job of recognizing same. Beyond that, fields seem to be specified by users when they define searches.
• Splunk has a simple ILM (Information Lifecycle management) story based on time. I didn’t probe for details.

Given its text search engine, Splunk does – well, it does text searches. And it stores searches, so they can be used for alerting or reporting. Indeed, Splunk persists and presumably updates results to stored searches, in a rough analog to materialized views.

Apparently, Splunk’s indexing is typically done via MapReduce jobs. I don’t know whether any actual Splunk searches are also done via MapReduce; surely they aren’t all, given the discussion of a near-real-time alerting engine and so on. Splunk fondly believes its MapReduce is an order of magnitude faster than SQL (I didn’t ask which SQL engines Splunk has in mind when they say this), and 5-10X faster than Hadoop. One efficiency trick is to look ahead and do Reduces in place where possible. This seems to be done automatically in the execution plan, ala Aster’s SQL-MapReduce, rather than having to be hand-coded. Splunk says its software can “easily” index 1-200 gigabytes of data per day on a commodity 8-core server, while maintaining an active search load, and 3-400 gigabytes are doable.

Splunk’s capabilities right now in tabular-style analytics seem to be limited to a command-line report builder, plus a GUI wizard that generates the command line. A few users have asked for support of third-party business intelligence tools, but Splunk hasn’t provided that yet. Nor can I find much evidence of ODBC/JDBC drivers for Splunk. But then, I have trouble understanding how Splunk could provide flexible and robust reporting unless it tokenized and indexed specific fields more aggressively than I think it now does.

]]> http://www.dbms2.com/2009/10/18/technical-introduction-to-splunk/feed/ 9 http://www.dbms2.com/2009/10/18/general-introduction-to-splunk/ http://www.dbms2.com/2009/10/18/general-introduction-to-splunk/#comments Sun, 18 Oct 2009 15:59:56 +0000 Curt Monash http://www.dbms2.com/?p=1119 I dropped by log analysis software vendor Splunk a few weeks ago for a chat with Marketing VP Steve Sommer (who some you may know from Cognos and/or Informix), Product Management VP Christina Noren, and above all co-founder/CTO Erik Swan. Splunk turns out to be a pretty interesting company, from both business and technical standpoints. For one thing, Splunk seems highly regarded by most people I mention it to.

Splunk’s technical stories include:

• Text search over log files.
• Business intelligence over text search. (That part sounds a lot like Attivio.)
• MapReduce with schema flexibility and smart multi-stage execution plans. (That part sounds a lot like Aster Data.)

More on those in a separate post.

Less technical Splunk highlights include:

• Splunk has ~1200 paying customers, and is adding a couple hundred more per quarter.
• Splunk has ~160 people.
• ~80% of Splunk sales are in North America.
• Typical Splunk sales prices are in the $10-50K range, with an average around $25K, or maybe that average is a bit over $30K. Some Splunk deals are six- or even seven-figure.
• Splunk is “quite profitable.”
• Splunk’s eponymous product is priced according to how much data is indexed per day. If you index half a gigabyte of logs per day or less, Splunk is completely free. So, while Splunk is closed-source, there’s something of an open-source-like Splunk adoption model.
• Splunk has been selling product for a couple of years. I gather Splunk 4 was recently released.
• Splunk’s biggest industry segments are, not too surprisingly,
  • Telco
  • Financial services
  • Government
  • “Online”
• Splunk’s paying customers seem to use it mainly for:
  • Web logs and associated network event logs (this seems to be the biggest area)
  • Security and perhaps other general IT log analysis
  • Physical security logs (mainly in the government)
  • Anti-fraud (I’m not sure how that works)
• One would think Splunk would be used to manage a lot of intelligence telemetry, but that wasn’t particularly hinted at.
• In general, the core problem Splunk is used for is log analysis for trouble-shooting purposes.
• Splunk’s nonpaying users are more diverse; examples mentioned included windmill operations and protein research.
• Splunk’s customers include Aster Data flagship accounts MySpace and LinkedIn. I bet many other top web companies are Splunk customers as well.

• Infobright now has >100 paying customers.
• Typical database size is from the low 100s of gigabytes to the low single-digit number of terabytes.
• Agile development is at or approaching two-week release cycles.
• Like Kickfire, Infobright  has a multi-year deal with MySQL that insulates it against many potential Oracle/MySQL shenanigans.
• From an industry perspective, Infobright’s customer base sounds a lot like other vendors’:
  • Data mart outsourcing/online analytics
  • Log files for websites
  • Telecommunications
  • Financial services
  • OEM, especially in the markets cited above
  • “Hey, we’re beginning to see the occasional energy deal”
  • A few random others
• Infobright is seeing some household-name customers, who surely have big-name analytic DBMS products, but who also have a policy that open source is the default choice, and if open source can get the job done then the favorite closed-source choices aren’t used.
• Infobright has the usual open-source community story — lots of involvement and engagement in the forums, but contributions are limited mainly to connectivity, utility scripts, etc. (Maybe some national language translation too; I’m not sure.)

• Some are in heavy production with Hadoop, and closely engaged with Cloudera. Others are active Hadoop users but are very secretive. Yet others signed up for initial Hadoop training last week.
• Some have Hadoop clusters in the thousands of nodes. Many have Hadoop clusters in the 50-100 node range. Others are just prototyping Hadoop use. And one seems to be “OEMing” a small Hadoop cluster in each piece of equipment sold.
• Many export data from Hadoop to a relational DBMS; many others just leave it in HDFS (Hadoop Distributed File System), e.g. with Hive as the query language, or in exactly one case Jaql.
• Some are household names, in web businesses or otherwise. Others seem to be pretty obscure.
• Industries include financial services, telecom (Asia only, and quite new), bioinformatics (and other research), intelligence, and lots of web and/or advertising/media.
• Application areas mentioned — and these overlap in some cases — include:
  • Log and/or clickstream analysis of various kinds
  • Marketing analytics
  • Machine learning and/or sophisticated data mining
  • Image processing
  • Processing of XML messages
  • Web crawling and/or text processing
  • General archiving, including of relational/tabular data, e.g. for compliance

We went over this list so quickly that we didn’t go into much detail on any one user. But one example that stood out was of an ad serving firm that had an “aggregation pipeline” consisting of 70-80 MapReduce jobs.

I also talked yesterday again w/ Omer Trajman of Vertica, who surprised me by indicating a high single-digit number of Vertica’s customers were in production with Hadoop — i.e., over 10% of Vertica’s production customers. (Vertica recently made its 100th sale, and of course not all those buyers are in production yet.) Vertica/Hadoop usage seems to have started in Vertica’s financial services stronghold — specifically in financial trading — with web analytics and the like coming on afterwards. Based on current prototyping efforts, Omer expects bioinformatics to be the third production market for Vertica/Hadoop, with telecommunications coming in fourth.

Unsurprisingly, the general Vertica/Hadoop usage model seems to be:

• Do something to the data in Hadoop
• Dump it into Vertica to be queried

What I did find surprising is that the data often isn’t reduced by this analysis, but rather exploded in size. E.g., a complete store of mortgage trading data might be a few terabytes in size, but Hadoop-based post processing can increase that by 1 or 2 orders of magnitude. (Analogies to the importance and magnitude of “cooked” data in scientific data processing come to mind.)

And finally, I talked to Aster a few days ago about the usage of its nCluster/Hadoop connector. Aster characterized Aster/Hadoop users’ Hadoop usage as being of the batch/ETL variety, which is the classic use case one concedes to Hadoop even if one believes that MapReduce should commonly be done right in the DBMS.

Related link

• An August, 2008 round-up of MapReduce applications.

Highlights of our visit included:

• There are dozens of people at Yahoo doing Hadoop development that will wind up getting open sourced. (Full-time or close to it.) In particular, everything Mark’s team does goes to open source.
• Yahoo is moving as much of its analytics to Hadoop as possible. Much of this is being moved away from Oracle and from Yahoo’s own Everest.
• A column store is being put on top of HDFS, based on Yahoo technology. Columns will be striped across nodes. Perhaps that’s why the effort is called Project Zebra.
• Mark believes that in a year Hadoop will be much further along in meeting traditional data warehousing requirements, in areas such as:
  • Metadata
  • SLAs/high availability/other workload management
  • Data retention policies
  • Security/privacy*
• Yahoo views the time-to-market benefits of Hadoop as being more important than TCO.

*I also spoke with a couple of Mark’s Yahoo colleagues, on his introduction, who are being less helpful than he is about clarifying what I am or am not allowed to say for publication. But I will say that I was heartened by the degree of concern they showed for doing the right thing with regard to privacy. I was not as heartened by the concrete ideas — or lack thereof — for making it happen. But frankly, I don’t think it’s a solvable technical problem. Rather, it should be a huge priority on the legal/political front.

We also talked some about Pig, Yahoo’s non-SQL DML (Data Manipulation Language) for Hadoop, which is however getting a SQL interface. And we talked about Pig vs. Hive. But I recently heard a rumor all that is in flux, so I won’t write it up now.

Mark sent along a couple of interesting slide presentations by a colleague. After some back and forth as to whether I could post them, he suggested I post these links to similar material instead.