• Google has published about a project called Dremel. Per Tasso Agyros, one of Dremel’s key concepts is nested data structures.
• Those arrays that the XLDB/SciDB folks keep talking about are meant to be nested data structures. Scientific data is of course log-oriented. eBay was very interested in that project too.
• Facebook’s log files have a big nested data structure flavor.

I don’t have a grasp yet on what exactly is happening here, but it’s something.

Jonathan’s core claims for Cassandra include:

• Cassandra is shared-nothing.
• Cassandra has good approaches to replication and partitioning, right out of the box.
• In particular, Cassandra is good for use cases that distribute a database around the world and want to access it at “local” latencies. (Indeed, Jonathan asserts that non-local replication is a significant non-big-data Cassandra use case.)
• Cassandra’s scale-out is application-transparent, unlike sharded MySQL’s.
• Cassandra is fast at both appends and range queries, which would be hard to accomplish in a pure key-value store.

In general, Jonathan positions Cassandra as being best-suited to handle a small number of operations at high volume, throughput, and speed. The rest of what you do, as far as he’s concerned, may well belong in a more traditional SQL DBMS. 

Further highlights of our talks included, as best I understood them:

• Cassandra is based in parts both on Google’s BigTable paper of 2006 and Amazon’s Dynamo paper of 2007.
  • The core of what Cassandra takes from BigTable is based on log-structured merge trees, which actually entered the computer science literature in 1996.
  • Cassandra’s approach to horizontal scaling, replication, failover, etc. seems to be based Dynamo.
• There seems to be a logical concept of “row” in Cassandra, or it’s at least meaningful to use the SQL/relational concept of a “row” when talking about Cassandra data. However, Cassandra is closer to being a column-based data store than a row-based one. (Not the same thing, but closer.)
• Even so, it only takes a single seek to return a whole Cassandra “row”.
• Cassandra writes data quite differently from the way a classical OLTP DBMS would.
  • Cassandra writes just the data elements – i.e., fields – that are actually being inserted or changed, not whole rows.
  • One benefit is that Cassandra data is very sparse. NULLs aren’t stored in any way, and hence in particular take up no space.
  • Another benefit – and one of the core concepts of Cassandra – is that you can implicitly assume different schemas for different rows of the same “table.” In particular, you can add data for columns that you didn’t envision when you first started storing “rows” of the same “table.”
  • Writes are collected into sorted “memtables,” which from time to time are sent to disk. Once data gets to disk, it’s immutable, except for occasional merge/reorganization/garbage collection.
    • Jonathan claims, plausibly, that this makes write throughput very fast (because the I/O is fundamentally sequential in nature.)
    • The default as to how long data typically stays in memory before it gets persisted to disk is “whichever comes first of {64 MB written, 300k updates, 1 hour}”.
  • Cassandra has durability – guaranteed non-loss of data – assuming fsync is turned on. fsync seems to create a 15% or so overhead.
  • However, Cassandra has no concept of a “transaction.”
  • As one would expect, data can be read even before it has been persisted to disk.
• According to Jonathan, Cassandra can do about 14,000 writes or 7,000 reads per second, on a quad-core server.
  • Those figures scale pretty linearly with the number of servers. (There’s some overhead for network latency.)
  • Those figures assume a five-column row.
  • Cassandra’s write-performance figures are only “mildly sensitive” to the width of the row. E.g., doubling row width only gives a 15-20% throughput hit, due to some fixed per-row overhead. That said, I imagine going 100X in row width would create a major slowdown, although perhaps while measuring width more in bytes than in column count.
  • Cassandra’s performance has been growing nicely in each point release. Jonathan thinks this general trend will continue.
• Jonathan thinks Cassandra is pretty good at keeping your data safe.
  • Each node has a commit log.
  • When a node goes down, its writes are buffered until it comes back up.
• You can run Hadoop MapReduce straight against Cassandra files.
• A Cassandra node might hold anything from 10s of gigabytes to multiple terabytes of data. You might want to go with the low end if you want to have lots of cache hits.
• Solid-state storage would speed up Cassandra reads, not writes, and is not widely used with Cassandra yet.
• Jonathan says Cassandra is really good at handling time series data, by which I suspect he means log files. Cloudkick is a user of this capability.

I certainly didn’t grasp everything about Cassandra replication and partitioning strategies. That wasn’t the focus of our talks, and anyway I got the impression they are so flexible that there’s little that can firmly be said about them. But I did get the impressions:

• You set your consistency rules in the Cassandra API, not on a per-table basis. (I think this means that a lack of administrative tools is supposedly a feature, not a drawback.)
• As a practical matter, Cassandra users commonly take one of two approaches to consistency:
  • RYW consistency, most commonly with N = 3 and R = W = 2.
  • Geographically dispersed eventual consistency.
• Cassandra data is most commonly distributed via consistent hashing, but other options are “pluggable.”
• If you add a node, the busiest note automagically decides to ship some data over, reducing its load. Of course, this only works if you get the new node on before the old node is so maxed out it doesn’t have time to do the shipping.

When we talked in March, the next release of Cassandra was going to be 0.7. Cassandra 0.7 was going to be a performance/scalability release, for example fixing the flaw that garbage collection read rows into memory one at a time. After that, Cassandra 0.8 was to be a feature release, with one planned feature being more automatic index management and/or materialized-view-like capability, so as to reduce the burden on Cassandra developers of schema management.

Related links

• My March NoSQL links post included the Google and Amazon papers
• The March 2, 2010 Cloudkick post also linked above goes into a lot of detail, including what they think is great about Cassandra and what they think is still missing

And it’s not going to get any better. Next week = the oft-postponed elder care trip. Then I’m back for a short week. Then I’m off on my quarterly visit to the SF area. Soon thereafter I’ve have a lot to do in connection with Enzee Universe. And at that point another month will have gone by.

Anyhow:

• Back in January, Oracle finally briefed me on Exadata 2. I also requested and got permission to post what I regarded as pretty interesting slides, then never got around to doing so. Well, here they are. (Pay no attention to the word “Confidential”.)
• Two people I have a lot of respect for, Cindi Howson and Doug Henschen, seem bullish on SAP’s in-memory NewDB efforts. But for a variety of execution reasons, I’m skeptical that this will matter for anything except SAP’s analytics suite. I.e., I don’t think anybody much except SAP will write OLTP apps to it, and I don’t think that without OLTP apps being written to it it’s much more than Business Objects’ answer to QlikView.
• I just learned that Netezza’s previous geospatial technology didn’t get ported to TwinFin. However, Netezza obviously found a geospatial alternative.

I ‘m beginning to make a habit of asking vendors for a postable version of their slide decks. Sybase IQ is another example.

• Google is doing something called BigQuery that is “SQL-like” for big data analytics. I don’t know anything about it.
• I also don’t know anything about IBM BigSheets yet. It sounds something like Datameer, but that could be way off the mark.

• ITA Software builds huge OLTP systems that it runs itself on behalf of airlines.
• Very, very unusually, ITA Software builds these huge OLTP systems in LISP.
• ITA Software is an Oracle shop (see Dan Weinreb’s comment).
• ITA Software is run by a techie (again, see Dan Weinreb’s comment).
• ITA Software has an interesting screen-scraping/web ETL project called Needlebase

ITA’s software does both price/reservation lookup/checking and reservation-making. I’ve had trouble keeping it straight, but I think the lookup is ITA’s actual business, and the reservation-making is ITA’s Next Big Thing. This is one of the ultimate federated-transaction-processing applications, because it involves coordinating huge OLTP systems run, in some cases, by companies that are bitter competitors with each other. Network latencies have to allow for intercontinental travel of the data itself.

Indeed, airline reservation systems are pretty much the OLTP ultimate in themselves. As the story goes, transaction monitors were pretty much invented for airline reservation systems in the 1960s.

A really small project for ITA Software is Needlebase. I stopped by ITA to look at Needlebase in January, and what it is is a very smart and hence interesting screen-scraping system. The idea is people publish database information to the web, and you may want to look at their web pages and recover the database records it is based on. Applications of this to the airline industry, which has 100s of 1000s of price changes per day — and I may be too low by one or two orders of magnitude when I say that — should be fairly obvious. ITA Software has aspirations of applying Needlebase to other sectors as well, or more precisely having users who do so. Last I looked, ITA hadn’t put significant resources behind stimulating Needlebase adoption — but Google might well change that.

Edit: I just re-found an old characterization of (some of) what ITA Software does by — who else? — Dan Weinreb:

I am working on our new product, an airline reservation system.  It’s an online transaction-processing system that must be up 99.99% of the time, maintaining maximum response time (e.g. on www.aircanada.com).  It’s a very, very complicated system.  The presentation layer is written in Java using conventional techniques.  The business rule layer is written in Common Lisp; about 500,000 lines of code (plus another 100,000 or so of open source libraries).  The database layer is Oracle RAC.  We operate our own data centers, some here in Massachusetts and a disaster-recovery site in Canada (separate power grid).

Related links

• ITA Software and Needlebase websites
• More about LISP

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

A system for large-scale processing of data, comprising:

• a plurality of processes executing on a plurality of interconnected processors;
• the plurality of processes including a master process, for coordinating a data processing job for processing a set of input data, and worker processes;
• the master process, in response to a request to perform the data processing job, assigning input data blocks of the set of input data to respective ones of the worker processes;
• each of a first plurality of the worker processes including an application-independent map module for retrieving a respective input data block assigned to the worker process by the master process and applying an application-specific map operation to the respective input data block to produce intermediate data values, wherein at least a subset of the intermediate data values each comprises a key/value pair, and wherein at least two of the first plurality of the worker processes operate simultaneously so as to perform the application-specific map operation in parallel on distinct, respective input data blocks;
• a partition operator for processing the produced intermediate data values to produce a plurality of intermediate data sets, wherein each respective intermediate data set includes all key/value pairs for a distinct set of respective keys, and wherein at least one of the respective intermediate data sets includes respective ones of the key/value pairs produced by a plurality of the first plurality of the worker processes;
• and each of a second plurality of the worker processes including an application-independent reduce module for retrieving data, the retrieved data comprising at least a subset of the key/value pairs from a respective intermediate data set of the plurality of intermediate data sets and applying an application-specific reduce operation to the retrieved data to produce final output data corresponding to the distinct set of respective keys in the respective intermediate data set of the plurality of intermediate data sets, and wherein at least two of the second plurality of the worker processes operate simultaneously so as to perform the application-specific reduce operation in parallel on multiple respective subsets of the produced intermediate data values.

The way a patent works is that you make a big claim and, just in case it’s later invalidated, you also make more specialized sub-claims. What’s more, in a software patent, you claim everything twice, once as a “system” and once as a “method.”

When a patent takes that long to issue and has a core claim that wordy, one can assume there was much back and forth with the PTO (Patent and Trademark Office) to whittle it down to something they felt they could approve. At a guess, I’d conjecture that the supposedly unique parts of the claim are concentrated in the areas I bolded above, and that the PTO doesn’t think the claim would be patentable unless most or all of them were included.

So should the claim have been approved even so? Let’s consider prior art. Oracle has long been able to parallelize ala MapReduce. I don’t see anything in the claim that isn’t preceded by what Oracle did, except maybe the emphasis on key/value pairs. (And the same statement applies to the other 15 claims in the patent, at least on a quick skim.) I forget the details of SenSage’s quasi-MapReduce, which also preceded the Google patent filing, but I imagine something similar would be true about it.

There is no doubt that Google popularized the ideas of MapReduce — which turns out to have been a worthy public service. In one great example of that popularization, the seminal paper on parallel data mining is almost laughable in how it deviates from MapReduce key/value pair formalism — but it still seems to have been inspired by Google’s MapReduce. But that’s a different matter; popularization != invention, even though there’s a certain connection between the two in patent law. Actually, Google also often does get credit for having “invented” MapReduce, including regrettably in the marketing materials of clients I can’t talk out of saying that and which now might be looking into the barrel of the Google patent (hello Aster); but again, saying something doesn’t make it enforceable in court.

So what it all boils down to is:

Should Google’s patent on the idea of parallelizing the handling of sets of application-visible key/value pairs be regarded as valid?

The United States PTO, which is paid to think about these things, has evidently decided Yes. I disagree. In simplest terms, my reason is that key/value pairs have been around for decades, and so:

Anything which was known or obvious without special reference to key/value pairs doesn’t suddenly become non-obvious when key/value pairs are mixed in.

If Google ever tries to enforce its MapReduce patent, I’m available as an expert witness for the other side.

Related links

• GigaOm and Ars Technica on the Google MapReduce patent
• Another silly software patent issue

• MapReduce was named and popularized — but not invented — by Google.
• “MapReduce” variously refers to:
  • A programming paradigm
  • Execution engines that implement the programming paradigm
  • Distributed file systems that work with the execution engines
• In particular, Hadoop is a MapReduce execution engine that includes or is closely associated with HDFS (Hadoop Distributed File System).
• MapReduce and analytic DBMS can interact in a number of different ways, including:
  • Tight integration between a DBMS and exposed MapReduce functionality, e.g. Aster Data’s SQL/MapReduce or Greenplum.
  • Integrated MapReduce “under the covers”, e.g. SenSage or Oracle. This may or may not follow all the rules Google laid out for MapReduce, but it’s at least similar in spirit.
  • Looser coupling between DBMS and a MapReduce system, e.g. Vertica/Hadoop, in which MapReduce may or may not run on a different cluster than the DBMS.
  • Not at all, except perhaps insofar as a quasi-DBMS such as Hive is implemented over a MapReduce system such as Hadoop/HDFS.
• As predicted by Monash’s First Law of Commercial Semantics, different vendors have individual variants on those themes. For example, as per a registration-required white paper, Splunk is moving to publicly expose a not-quite-complete form of MapReduce.
• MapReduce implementations such as Hadoop are sometimes regarded as part of the NoSQL “movement”. When they are, many generalities about NoSQL — such as that it doesn’t deal with analytics — are falsified.
• So far as I can tell, mainstream enterprise (as opposed to web, scientific, investment, etc.) data mining folks may be looking at MapReduce for data mining, but they haven’t done much to adopt it yet. Probably that’s because the outfits who have the greatest need are the same ones that have the largest sunk investments in more traditional ways of doing data mining.
• Cloudera != Hadoop. On the other hand, if you want to use Hadoop, it makes a lot of sense to do business with Cloudera.
• Non-DBMS MapReduce != Hadoop. On the other hand, Hadoop is the default choice for non-DBMS MapReduce.
• MapReduce != Hadoop, period. DBMS-based MapReduce is also a legitimate technical strategy.

• MapReduce is something very new
• MapReduce involves strict adherence to the Map-Reduce programming paradigm
• MapReduce is a single technology

So let’s give it a try.

When Dave DeWitt and Mike Stonebraker leveled their famous blast at MapReduce, many people thought they overstated their case. But one part of their story – one that both Mike and Dave say was most central to their case – was never effectively refuted, namely the claim that these ideas aren’t particularly new. I haven’t actually read enough computer science literature to have an independent opinion on that issue. But I’ll say this – claims from companies such as SenSage, Oracle, or Splunk that “We’ve been doing MapReduce all along” seem pretty credible to me.

True, what those companies were doing things may not have looked exactly like the instant-classic MapReduce programming paradigm. But the same is true of many things almost everybody would agree count as MapReduce. In particular, it is often not the case that you alternate Map and Reduce steps, each of whose outputs is a set of simple <Key, Value> pairs, with data redistributed based on Key at every step.

Here are some examples of what I mean, drawn from my recent MapReduce webinar.

• If you do text indexing in MapReduce, your goal is to wind up with a text index. So at some point you Reduce to a pair <WordName, {all the (DocumentID, offset) pairs for the whole corpus, suitably ordered}>. That’s a heckuva compound “Value”.
• The goal of data mining is usually to estimate a rather small number of parameters based on a large overall data set, often – depending on algorithm – in the form of a single vector. When you do that in MapReduce. you partition data among nodes, calculate something on each node that is structured more or less like your final vector. So when it comes time for the reduce, you just ship all of your vectors – one per node – to a single Reduce node, and do the appropriate math. Redistribution based on Key would be quite pointless.
• When you sessionize clickstream logs in MapReduce, you may have just as many output records as input records. However, they now are reformatted, and might have a SessionID appended. In those cases, Reduce isn’t doing much by the way of reduction.
• And as I happens in some Vertica-Hadoop use cases around mortgage trading, sometimes MapReduce can even make data sets vastly larger.

By no means do I think this is a weakness of the MapReduce programming paradigm. Rather, I think it’s a MapReduce strength. But it’s not quite the way MapReduce has been promoted and explained to the IT public.

Finally: MapReduce, as commonly conceived, spans two different – albeit closely related – technology domains:

• Parallel programming
• Distributed data management

For example, I imagine Greenplum’s and Vertica’s MapReduce/SQL combined syntaxes are very similar to each others. But Vertica’s data management implementation of MapReduce, which relies on Hadoop, is very different from Greenplum’s, which is tied into the Greenplum DBMS. Similary, non-DBMS MapReduce implementations are commonly associated with distributed file systems – notably HDFS (Hadoop Distributed File Systems) or Google’s internal GFS (Google File System). In those systems, the parallel language execution part should be aware of how the distributed file management part works – but perhaps that awareness can be pretty lightweight.

Right now, this is a distinction pretty much without a difference. If you choose an implementation of MapReduce — like pure Hadoop (say in the Cloudera distribution) or Hadoop-Vertica or Aster Data’s SQL/MapReduce – you’re basically picking an entire technology stack. But those stacks are going to do a whole lot of changing and maturing in the near future – and as they do, it’s likely that projects will interact or even combine in all sorts of interesting ways.

Bottom line: There are a lot of different ways to exploit MapReduce-related technology.

What Fusion Tables really seems to be is a spreadsheet without the formulae. That is, it’s a place to dump data in a grid of cells, comment on it, version it, and do elementary data manipulation.  This could, I guess, be useful as an alternative to traditional RDBMS — assuming, of course, that you want to have a row-by-row debate about 100 megs of data.

Seriously, while Google Fusion Tables bears some vague resemblance to what I’m thinking about for the future of both business intelligence and data marts, it sounds as if it has a long way to go before it’s something most enterprises should spend time looking at.

Natural language and classic science fiction

Actually, there’s a pretty well-known example of BI near-perfection — the Star Trek computers, usually voiced by the late Majel Barrett Roddenberry. They didn’t have a big role in the recent movie, which was so fast-paced nobody had time to analyze very much, but were a big part of the Star Trek universe overall. Star Trek’s computers integrated analytics, operations, and authentication, all with a great natural language/voice interface and visual displays. That example is at the heart of a 1998 article on natural language recognition I just re-posted.

As for reality: For decades, dating back at least to Artificial Intelligence Corporation’s Intellect, there have been offerings that provided “natural language” command, control, and query against otherwise fairly ordinary analytic tools. Such efforts have generally fizzled, for reasons outlined at the link above. Wolfram Alpha is the latest try; fortunately for its prospects, natural language is really only a small part of the Wolfram Alpha story.

A second theme has more recently emerged — using text indexing to get at data more flexibly than a relational schema would normally allow, either by searching on data values themselves (stressed by Attivio) or more by searching on the definitions of pre-built reports (the Google OneBox story). SAP’s Explorer is the latest such view, but I find Doug Henschen’s skepticism about SAP Explorer more persuasive than Cindi Howson’s cautiously favorable view. Partly that’s because I know SAP (and Business Objects); partly it’s because of difficulties such as those I already noted.

Flexibility and data exploration

It’s a truism that each generation of dashboard-like technology fails because it’s too inflexible. Users are shown the information that will provide them with the most insight. They appreciate it at first. But eventually it’s old hat, and when they want to do something new, the baked-in data model doesn’t support it.

The latest attempts to overcome this problem lie in two overlapping trends — cool data exploration/visualization tools, and in-memory analytics. Tableau and Spotfire are known more for the former; hot BI vendor QlikTech is known for both. And many vendors — established or otherwise — are going to in-memory OLAP.

Collaboration and communication

The reason I’m finally buckling down and posting on this subject is the announcement of Google Wave, which I think foreshadows a revolution in communication and collaboration technology. Google Wave augurs two primary advances. First, it shows how to make email, instant messaging, microblogging, and so on much more useful. Second, Google Wave could evolve in a way that — finally — makes it truly practical for end-users to set up ad-hoc mini-portals that combine arbitrary URL-possessing resources, exposed to arbitrary workgroups of people.

If and when both of those promises are fulfilled, it will become vastly easier for people to reason together about analytic questions. That may take a little while, as Google Wave obviously wasn’t designed with business intelligence in mind. But whether from Google or from a frightened Microsoft redoubling its SharePoint efforts, there’s hope that we’ll see a leap forward in general collaboration technology. And since BI vendors are doing a generally decent job of exposing queries, charts and so on as portlets, it seems likely that business intelligence will benefit from the collaboration arms race.

That’s important. The first time I heard that reporting was as important for communication as for analytics was from Pilot Software a quarter-century or so ago, and it’s just as true now as it was then. In its first incarnations it probably will be a little too dumb for my tastes, focusing more on mindless reporting and same-old KPIs than on deeper analysis. Still, it’s a move in a good direction.

Other directions

As I said at the beginning, I find it too daunting to try to cover all facets of this subject in one post. So I’ll leave out, at a minimum:

• Data warehousing performance and TCO, which I of course write about extensively
• Complex event/stream processing, which I’ve written quite a bit about too
• Data mining and predictive analytics
• Operational BI

plus some hobby horses you probably don’t want to hear about anyway until I work out a better way of articulating my opinions.

But by all means please comment on what I’ve left out just as vigorously as on what I’ve included. This post is just the first of many to come.

]]> http://www.dbms2.com/2009/05/30/reinventing-business-intelligence/feed/ 15