• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

There’s much confusion about the management of text data, among technology users, vendors, and investors alike. Reasons seems to include:

• The terminology around text data is inaccurate.
• Data volume estimates for text are misleading.
• Multiple different technologies are in the mix, including:
  • Enterprise text search.
  • Text analytics — text mining, sentiment analysis, etc.
  • Document stores — e.g. document-oriented NoSQL, or MarkLogic.
  • Log management and parsing — e.g. Splunk.
  • Text archiving — e.g., various specialty email archiving products I couldn’t even name.
  • Public web search — Google et al.
• Text search vendors have disappointed, especially technically.
• Text analytics vendors have disappointed, especially financially.
• Other analytic technology vendors ignore what the text analytic vendors actually have accomplished, and reinvent inferior wheels rather than OEM the state of the art.

Above all: The use cases for text data vary greatly, just as the use cases for simply-structured databases do.

There are probably fewer people now than there were six years ago who need to be told that text and relational database management are very different things. Other misconceptions, however, appear to be on the rise. Specific points that are commonly overlooked include:

• The terms “unstructured” or “semi-structured” data are inherently misleading. That’s why I favor “multi-structured” or “poly-structured” instead. (“Multi-structured” seems to be winning; e.g., it’s been adopted by Teradata and Teradata/Aster.)
• The “social media” text data any one enterprise brings in house isn’t all that much. For example, Attensity serves many different enterprises’ social media needs from a single 20-terabyte data store, and reports that no single enterprise has required as much as 1 terabyte of text yet. Text data may consume a lot of storage on spinning disks somewhere, but it’s not that big a factor in future DBMS industry growth. (That 20 terabyte figure does seem low.)
• Structured databases are typically worth a lot more per bit than other kinds. The most valuable electronic data, per-bit, is probably records of significant economic transactions — purchases, sales, money transfers, etc. The least valuable may be sensor log files, whose contents consist mainly of “Nothing going on here; ping you again in a minute.” Email logs, web interaction data and many other kinds fall somewhere in between. Highly valuable documents — such as signed contracts — generally persist in paper as well as electronic forms. Investors commonly overlook this point.
• The enterprise text search industry is screwed up.
  • FAST was a goofy company before it was acquired for far too much money by Microsoft.
  • Autonomy was a goofy company before it was acquired for far too much money by HP.
  • Google’s enterprise efforts are quiet.
  • The integration of text search and relational DBMS — e.g. at Oracle — has languished, with poor performance and evident lack of management attention.
  • Smaller text search vendors don’t seem to be getting a lot of traction — e.g., Coveo has a decent reputation, but when’s the last time you heard much about them? What has Attivio actually accomplished?
• Text analytics is a small business. Add up the revenue for Attensity, Clarabridge, Lexalytics, Temis, and all the others, and you might poke above $100 million, especially now that Attensity had a three-way merger. Then again, you might not.
• Even so, the text analytics vendors have developed sophisticated technology. In particular, you can use it to get a pretty good idea as to what people are writing about you, individually or as groups.

• HP hired a software guy, Leo Apotheker, as CEO, and a software guy with a liking for high-end services, Ray Lane, as chairman. Now a Leo Apotheker conference call suggests HP will increase its emphasis on software, and maybe high-end services as well. No surprise. The article suggests, however,  that HP at this point has no clear strategy along these lines. That’s no surprise either.
  • And then there’s Sarah Lacy’s take, of which the interesting part reads “Separately, Andreessen has said that he thinks enterprise software is ripe for disruption and his firm is going to fund a new generation of Oracle-killers.”
  • I added more on Ray Lane’s tenure at Oracle over on Software Memories.
• Netezza had a falling out with its original supplier of geospatial technology, Intelligent Integration Systems (IISi), and a lawsuit ensued over alleged copying. Now ISSi has upped the stakes, essentially alleging that Netezza’s new geospatial software doesn’t work, and that hence the CIA (evidently a Netezza user) is killing the wrong people via drone strikes. Netezza has wisely selected from its short list of acceptable responses, including versions of:
  • “All our classified customers are happy, and if we told you anything more than that, that would kind of defeat the purpose of being classified, wouldn’t it?”
  • “Copy, schmopy. A polygon is a polygon, and has been since Euclid.”
  • “We don’t have no steenking bugs.”
• OCZ, whoever they are, are trying to offer solid-state drives with PCIe-like bandwidth, which makes sense in that most observers except Teradata think the SAS interface isn’t fast enough for solid-state.
• Speaking of Teradata, I’d been wondering somewhat as to why they just shut down Kickfire’s product line after acquiring its assets. Well, somebody who tested a Kickfire box told me that — great TPC-H results notwithstanding — it turned out not to be nearly as fast as one might think, on real-life data sets that didn’t fit entirely into RAM. Hard though such a thing may be to imagine, it turns out that Kickfire’s TPC-H results were yet less significant than I thought they were.
• I haven’t been looking at High Scalability nearly as much as I should, and that’s an understatement. It’s an outstanding blog.
• A couple of Google execs offered some predictions about the future of online advertising, which might be of interest to anybody selling analytic (or text analytic) technology to the online/digital media market.
• The BBC shows us what a single 133-character tweet plus its metadata look like in JSON. (All 1582 characters.)
• Huffington Post’s CEO made some comments about influencers which are additive to what I’ve been saying about influencers over on Strategic Messaging. (If you don’t read that — well, it’s my blog about marketing.)
  
• Speaking of my other blogs, I’m not bothering to put up a separate post like this over on Text Technologies, where thee posts I have put up recently tend to be (at least by my standards) relatively link-heavy anyway, but I have a couple more to share even so:
  • Paul Carr’s 7 rules for TechCrunch/AOL employees are really funny.
  • Some major search engine marketing experts are sounding defeatist about web spam.

• Google has published about a project called Dremel. Per Tasso Agyros, one of Dremel’s key concepts is nested data structures.
• Those arrays that the XLDB/SciDB folks keep talking about are meant to be nested data structures. Scientific data is of course log-oriented. eBay was very interested in that project too.
• Facebook’s log files have a big nested data structure flavor.

I don’t have a grasp yet on what exactly is happening here, but it’s something.

Jonathan’s core claims for Cassandra include:

• Cassandra is shared-nothing.
• Cassandra has good approaches to replication and partitioning, right out of the box.
• In particular, Cassandra is good for use cases that distribute a database around the world and want to access it at “local” latencies. (Indeed, Jonathan asserts that non-local replication is a significant non-big-data Cassandra use case.)
• Cassandra’s scale-out is application-transparent, unlike sharded MySQL’s.
• Cassandra is fast at both appends and range queries, which would be hard to accomplish in a pure key-value store.

In general, Jonathan positions Cassandra as being best-suited to handle a small number of operations at high volume, throughput, and speed. The rest of what you do, as far as he’s concerned, may well belong in a more traditional SQL DBMS. 

Further highlights of our talks included, as best I understood them:

• Cassandra is based in parts both on Google’s BigTable paper of 2006 and Amazon’s Dynamo paper of 2007.
  • The core of what Cassandra takes from BigTable is based on log-structured merge trees, which actually entered the computer science literature in 1996.
  • Cassandra’s approach to horizontal scaling, replication, failover, etc. seems to be based Dynamo.
• There seems to be a logical concept of “row” in Cassandra, or it’s at least meaningful to use the SQL/relational concept of a “row” when talking about Cassandra data. However, Cassandra is closer to being a column-based data store than a row-based one. (Not the same thing, but closer.)
• Even so, it only takes a single seek to return a whole Cassandra “row”.
• Cassandra writes data quite differently from the way a classical OLTP DBMS would.
  • Cassandra writes just the data elements – i.e., fields – that are actually being inserted or changed, not whole rows.
  • One benefit is that Cassandra data is very sparse. NULLs aren’t stored in any way, and hence in particular take up no space.
  • Another benefit – and one of the core concepts of Cassandra – is that you can implicitly assume different schemas for different rows of the same “table.” In particular, you can add data for columns that you didn’t envision when you first started storing “rows” of the same “table.”
  • Writes are collected into sorted “memtables,” which from time to time are sent to disk. Once data gets to disk, it’s immutable, except for occasional merge/reorganization/garbage collection.
    • Jonathan claims, plausibly, that this makes write throughput very fast (because the I/O is fundamentally sequential in nature.)
    • The default as to how long data typically stays in memory before it gets persisted to disk is “whichever comes first of {64 MB written, 300k updates, 1 hour}”.
  • Cassandra has durability – guaranteed non-loss of data – assuming fsync is turned on. fsync seems to create a 15% or so overhead.
  • However, Cassandra has no concept of a “transaction.”
  • As one would expect, data can be read even before it has been persisted to disk.
• According to Jonathan, Cassandra can do about 14,000 writes or 7,000 reads per second, on a quad-core server.
  • Those figures scale pretty linearly with the number of servers. (There’s some overhead for network latency.)
  • Those figures assume a five-column row.
  • Cassandra’s write-performance figures are only “mildly sensitive” to the width of the row. E.g., doubling row width only gives a 15-20% throughput hit, due to some fixed per-row overhead. That said, I imagine going 100X in row width would create a major slowdown, although perhaps while measuring width more in bytes than in column count.
  • Cassandra’s performance has been growing nicely in each point release. Jonathan thinks this general trend will continue.
• Jonathan thinks Cassandra is pretty good at keeping your data safe.
  • Each node has a commit log.
  • When a node goes down, its writes are buffered until it comes back up.
• You can run Hadoop MapReduce straight against Cassandra files.
• A Cassandra node might hold anything from 10s of gigabytes to multiple terabytes of data. You might want to go with the low end if you want to have lots of cache hits.
• Solid-state storage would speed up Cassandra reads, not writes, and is not widely used with Cassandra yet.
• Jonathan says Cassandra is really good at handling time series data, by which I suspect he means log files. Cloudkick is a user of this capability.

I certainly didn’t grasp everything about Cassandra replication and partitioning strategies. That wasn’t the focus of our talks, and anyway I got the impression they are so flexible that there’s little that can firmly be said about them. But I did get the impressions:

• You set your consistency rules in the Cassandra API, not on a per-table basis. (I think this means that a lack of administrative tools is supposedly a feature, not a drawback.)
• As a practical matter, Cassandra users commonly take one of two approaches to consistency:
  • RYW consistency, most commonly with N = 3 and R = W = 2.
  • Geographically dispersed eventual consistency.
• Cassandra data is most commonly distributed via consistent hashing, but other options are “pluggable.”
• If you add a node, the busiest note automagically decides to ship some data over, reducing its load. Of course, this only works if you get the new node on before the old node is so maxed out it doesn’t have time to do the shipping.

When we talked in March, the next release of Cassandra was going to be 0.7. Cassandra 0.7 was going to be a performance/scalability release, for example fixing the flaw that garbage collection read rows into memory one at a time. After that, Cassandra 0.8 was to be a feature release, with one planned feature being more automatic index management and/or materialized-view-like capability, so as to reduce the burden on Cassandra developers of schema management.

Related links

• My March NoSQL links post included the Google and Amazon papers
• The March 2, 2010 Cloudkick post also linked above goes into a lot of detail, including what they think is great about Cassandra and what they think is still missing

And it’s not going to get any better. Next week = the oft-postponed elder care trip. Then I’m back for a short week. Then I’m off on my quarterly visit to the SF area. Soon thereafter I’ve have a lot to do in connection with Enzee Universe. And at that point another month will have gone by.

Anyhow:

• Back in January, Oracle finally briefed me on Exadata 2. I also requested and got permission to post what I regarded as pretty interesting slides, then never got around to doing so. Well, here they are. (Pay no attention to the word “Confidential”.)
• Two people I have a lot of respect for, Cindi Howson and Doug Henschen, seem bullish on SAP’s in-memory NewDB efforts. But for a variety of execution reasons, I’m skeptical that this will matter for anything except SAP’s analytics suite. I.e., I don’t think anybody much except SAP will write OLTP apps to it, and I don’t think that without OLTP apps being written to it it’s much more than Business Objects’ answer to QlikView.
• I just learned that Netezza’s previous geospatial technology didn’t get ported to TwinFin. However, Netezza obviously found a geospatial alternative.

I ‘m beginning to make a habit of asking vendors for a postable version of their slide decks. Sybase IQ is another example.

• Google is doing something called BigQuery that is “SQL-like” for big data analytics. I don’t know anything about it.
• I also don’t know anything about IBM BigSheets yet. It sounds something like Datameer, but that could be way off the mark.

• ITA Software builds huge OLTP systems that it runs itself on behalf of airlines.
• Very, very unusually, ITA Software builds these huge OLTP systems in LISP.
• ITA Software is an Oracle shop (see Dan Weinreb’s comment).
• ITA Software is run by a techie (again, see Dan Weinreb’s comment).
• ITA Software has an interesting screen-scraping/web ETL project called Needlebase

ITA’s software does both price/reservation lookup/checking and reservation-making. I’ve had trouble keeping it straight, but I think the lookup is ITA’s actual business, and the reservation-making is ITA’s Next Big Thing. This is one of the ultimate federated-transaction-processing applications, because it involves coordinating huge OLTP systems run, in some cases, by companies that are bitter competitors with each other. Network latencies have to allow for intercontinental travel of the data itself.

Indeed, airline reservation systems are pretty much the OLTP ultimate in themselves. As the story goes, transaction monitors were pretty much invented for airline reservation systems in the 1960s.

A really small project for ITA Software is Needlebase. I stopped by ITA to look at Needlebase in January, and what it is is a very smart and hence interesting screen-scraping system. The idea is people publish database information to the web, and you may want to look at their web pages and recover the database records it is based on. Applications of this to the airline industry, which has 100s of 1000s of price changes per day — and I may be too low by one or two orders of magnitude when I say that — should be fairly obvious. ITA Software has aspirations of applying Needlebase to other sectors as well, or more precisely having users who do so. Last I looked, ITA hadn’t put significant resources behind stimulating Needlebase adoption — but Google might well change that.

Edit: I just re-found an old characterization of (some of) what ITA Software does by — who else? — Dan Weinreb:

I am working on our new product, an airline reservation system.  It’s an online transaction-processing system that must be up 99.99% of the time, maintaining maximum response time (e.g. on www.aircanada.com).  It’s a very, very complicated system.  The presentation layer is written in Java using conventional techniques.  The business rule layer is written in Common Lisp; about 500,000 lines of code (plus another 100,000 or so of open source libraries).  The database layer is Oracle RAC.  We operate our own data centers, some here in Massachusetts and a disaster-recovery site in Canada (separate power grid).

Related links

• ITA Software and Needlebase websites
• More about LISP

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

A system for large-scale processing of data, comprising:

• a plurality of processes executing on a plurality of interconnected processors;
• the plurality of processes including a master process, for coordinating a data processing job for processing a set of input data, and worker processes;
• the master process, in response to a request to perform the data processing job, assigning input data blocks of the set of input data to respective ones of the worker processes;
• each of a first plurality of the worker processes including an application-independent map module for retrieving a respective input data block assigned to the worker process by the master process and applying an application-specific map operation to the respective input data block to produce intermediate data values, wherein at least a subset of the intermediate data values each comprises a key/value pair, and wherein at least two of the first plurality of the worker processes operate simultaneously so as to perform the application-specific map operation in parallel on distinct, respective input data blocks;
• a partition operator for processing the produced intermediate data values to produce a plurality of intermediate data sets, wherein each respective intermediate data set includes all key/value pairs for a distinct set of respective keys, and wherein at least one of the respective intermediate data sets includes respective ones of the key/value pairs produced by a plurality of the first plurality of the worker processes;
• and each of a second plurality of the worker processes including an application-independent reduce module for retrieving data, the retrieved data comprising at least a subset of the key/value pairs from a respective intermediate data set of the plurality of intermediate data sets and applying an application-specific reduce operation to the retrieved data to produce final output data corresponding to the distinct set of respective keys in the respective intermediate data set of the plurality of intermediate data sets, and wherein at least two of the second plurality of the worker processes operate simultaneously so as to perform the application-specific reduce operation in parallel on multiple respective subsets of the produced intermediate data values.

The way a patent works is that you make a big claim and, just in case it’s later invalidated, you also make more specialized sub-claims. What’s more, in a software patent, you claim everything twice, once as a “system” and once as a “method.”

When a patent takes that long to issue and has a core claim that wordy, one can assume there was much back and forth with the PTO (Patent and Trademark Office) to whittle it down to something they felt they could approve. At a guess, I’d conjecture that the supposedly unique parts of the claim are concentrated in the areas I bolded above, and that the PTO doesn’t think the claim would be patentable unless most or all of them were included.

So should the claim have been approved even so? Let’s consider prior art. Oracle has long been able to parallelize ala MapReduce. I don’t see anything in the claim that isn’t preceded by what Oracle did, except maybe the emphasis on key/value pairs. (And the same statement applies to the other 15 claims in the patent, at least on a quick skim.) I forget the details of SenSage’s quasi-MapReduce, which also preceded the Google patent filing, but I imagine something similar would be true about it.

There is no doubt that Google popularized the ideas of MapReduce — which turns out to have been a worthy public service. In one great example of that popularization, the seminal paper on parallel data mining is almost laughable in how it deviates from MapReduce key/value pair formalism — but it still seems to have been inspired by Google’s MapReduce. But that’s a different matter; popularization != invention, even though there’s a certain connection between the two in patent law. Actually, Google also often does get credit for having “invented” MapReduce, including regrettably in the marketing materials of clients I can’t talk out of saying that and which now might be looking into the barrel of the Google patent (hello Aster); but again, saying something doesn’t make it enforceable in court.

So what it all boils down to is:

Should Google’s patent on the idea of parallelizing the handling of sets of application-visible key/value pairs be regarded as valid?

The United States PTO, which is paid to think about these things, has evidently decided Yes. I disagree. In simplest terms, my reason is that key/value pairs have been around for decades, and so:

Anything which was known or obvious without special reference to key/value pairs doesn’t suddenly become non-obvious when key/value pairs are mixed in.

If Google ever tries to enforce its MapReduce patent, I’m available as an expert witness for the other side.

Related links

• GigaOm and Ars Technica on the Google MapReduce patent
• Another silly software patent issue

• MapReduce was named and popularized — but not invented — by Google.
• “MapReduce” variously refers to:
  • A programming paradigm
  • Execution engines that implement the programming paradigm
  • Distributed file systems that work with the execution engines
• In particular, Hadoop is a MapReduce execution engine that includes or is closely associated with HDFS (Hadoop Distributed File System).
• MapReduce and analytic DBMS can interact in a number of different ways, including:
  • Tight integration between a DBMS and exposed MapReduce functionality, e.g. Aster Data’s SQL/MapReduce or Greenplum.
  • Integrated MapReduce “under the covers”, e.g. SenSage or Oracle. This may or may not follow all the rules Google laid out for MapReduce, but it’s at least similar in spirit.
  • Looser coupling between DBMS and a MapReduce system, e.g. Vertica/Hadoop, in which MapReduce may or may not run on a different cluster than the DBMS.
  • Not at all, except perhaps insofar as a quasi-DBMS such as Hive is implemented over a MapReduce system such as Hadoop/HDFS.
• As predicted by Monash’s First Law of Commercial Semantics, different vendors have individual variants on those themes. For example, as per a registration-required white paper, Splunk is moving to publicly expose a not-quite-complete form of MapReduce.
• MapReduce implementations such as Hadoop are sometimes regarded as part of the NoSQL “movement”. When they are, many generalities about NoSQL — such as that it doesn’t deal with analytics — are falsified.
• So far as I can tell, mainstream enterprise (as opposed to web, scientific, investment, etc.) data mining folks may be looking at MapReduce for data mining, but they haven’t done much to adopt it yet. Probably that’s because the outfits who have the greatest need are the same ones that have the largest sunk investments in more traditional ways of doing data mining.
• Cloudera != Hadoop. On the other hand, if you want to use Hadoop, it makes a lot of sense to do business with Cloudera.
• Non-DBMS MapReduce != Hadoop. On the other hand, Hadoop is the default choice for non-DBMS MapReduce.
• MapReduce != Hadoop, period. DBMS-based MapReduce is also a legitimate technical strategy.