Interesting quotes include:

WibiData is designed to store … transactional data side-by-side with profile and other derived data attributes.

… the ability to add new ad-hoc columns to a table enables more flexible analysis: output data that is the result of one analytic pipeline is stored adjacent to its input data, meaning that you can easily use this as input to second- or third-order derived data as well.

schemas can vary over time; you can easily add a field to a record, or delete a field. … But even though you start collecting that new data, your existing analysis pipelines can treat records like they always did; programs that don’t yet know about the new cookie are still compatible with both the old records already collected, and the new records with the additional field. New programs fill in default values for old data recorded before a field was added, applying the new schema at read time.

schemas for every column are stored in a data dictionary that matches column names with their schemas, as well as human-readable descriptions of the data.

Interesting aspects of the post that don’t lend themselves as well to being excerpted include:

• How the Produce-Gather “analysis calculus” — i.e. framework — works.
• How this all ties into Apache projects (and sub-projects) such as Hadoop, HBase, and Avro.

• Splunk is first and foremost an analytic DBMS …
• … used to manage logs and similar multistructured data.
• Splunk’s DML (Data Manipulation Language) is based on text search, not on SQL.
• Splunk has extended its DML in natural ways (e.g., you can use it to do calculations and even some statistics).
• Splunk bundles some (very) basic, Splunk-specific business intelligence capabilities.
• The paradigmatic use of Splunk is to monitor IT operations in real time. However:
  • There also are plenty of non-real-time uses for Splunk.
  • Splunk is proudest of its growth in non-IT quasi-real-time uses, such as the marketing side of web operations.

As in any release, a lot of Splunk 4.3 is about “Oh, you didn’t have that before?” features and Bottleneck Whack-A-Mole performance speed-up. One performance enhancement is Bloom filters, which are a very hot topic these days. More important is a switch from Flash to HTML5, so as to accommodate mobile devices with less server-side rendering. Splunk reports that its users — especially the non-IT ones — really want to get Splunk information on the tablet devices. While this somewhat contradicts what I wrote a few days ago pooh-poohing mobile BI, let me hasten to point out:

• Splunk is used for a lot of (quasi) real-time monitoring.
• Splunk’s desktop user interfaces are, by BI standards, quite primitive.

That’s pretty much the ideal scenario for mobile BI: Timeliness matters and prettiness doesn’t.

Hmm. Maybe StreamBase LiveView needs a mobile option as well …

Splunk’s basic use is to take the text string that is a log and make sense of it. But Splunk now also supports JSON structures. It does this via something called spath, which as you might guess from the name has XPath similarities. That probably bore more discussion than we found the time to have.

By the way: If you’re interested in BI over XML, that’s what my former clients at Skytide were founded to do, before they pivoted a bit. I don’t think those capabilities have disappeared from the product.

Splunk has graciously allowed me to post a slide deck. More stuff in there, including quotes from a customer — Expedia — that has 2700 Splunk users.

• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

• It’s called Odiago, and is one of my gratifyingly more numerous tiny clients.
• Odiago’s product line is called WibiData, after the justly popular We Be Sushi restaurants.
• We’ve agreed on a split exclusive de-stealthing launch. You can read about the company/founder/investor stuff on TechCrunch. But this is the place for — well, for the tech crunch.

WibiData is designed for management of, investigative analytics on, and operational analytics on consumer internet data, the main examples of which are web site traffic and personalization and their analogues for games and/or mobile devices. The core WibiData technology, built on HBase and Hadoop,* is a data management and analytic execution layer. That’s where the secret sauce resides. Also included are:

• REST APIs for interactive access.
• Import/export tools, including JDBC access.
• Management tools.
• Analytic libraries — data mining, predictive analytics, machine learning, and so on.

The whole thing is in beta, with about three (paying) beta customers.

*And Avro and so on.

The core ideas of WibiData include:

• ALL data pertaining to a single user (or mobile device) is kept in a single, possibly very long, HBase row.
• There are two primary operators in WibiData, Produce and Gather.
  • Produce operates on single rows. It can operate on one row at HBase speed (milliseconds) if you need to inform an interactive user response. Or it can operate on the whole database in batch via Hadoop MapReduce.
  • It is reasonable to think of Produce as mainly doing two things. One is the aforementioned serving of data out of WibiData into interactive applications. The other is scoring, classifying, recommending, etc. on individual users (i.e. rows), in line with an analytic model.
  • Gather typically operates on all your rows at once, and emits suitable input for a MapReduce Reduce step. It is reasonable to think of Gather as being a key cog in the training of analytic models.
• HBase schema management is done at the WibiData system level, not directly in applications. There’s a WibiData HBase data dictionary, powered by a set of system tables, that specifies cell data types/record types and, in effect, primitive schemas.

WibiData-enhanced HBase differs from relational DBMS in most of the ways you would imagine, both good and bad. In particular:

• Depending on how you look at it, WibiData-enhanced HBase either has no DML (Data Manipulation Language) at all, or else has one that ‘s a lot less rich than SQL.
• WibiData-enhanced HBase schemas are much more dynamic than SQL schemas.
• WibiData-enhanced HBase schemas can have nested or recursive data structures, such as array-valued cells.

To expand on each of those points in turn:

WibiData’s underlying one-giant-table philosophy notwithstanding, there are times you manage multiple tables with it. (For example, you ingest data into WibiData however you can, and then run transformations — typically batch — until the data is in the preferred structure.) While Wibidata does have ways to simulate joins, foreign keys, and so on, there’s nothing resembling referential integrity or foreign key constraints.

WibiData takes single-table schema flexibility to an extreme. Not only can different rows in the same table have different associated columns — something that relational systems can in effect also do via NULL values — but schemas can even change over the life of a column. If you have an array-valued cell storing the results of a marketing campaign, and you start recording more data partway through the campaign, then different rows in the table will, in the same column, hold different-sized arrays.

That nesting can also get pretty serious; where you’d have a single value in a relational table, you might have the equivalent of a whole relational table (or at least selection/view) in WibiData-enhanced HBase. For example, if a user visits the same web page ten times, and each time 50 attributes are recorded (including a timestamp), all 500 data – to use the word “data” in its original “plural of datum” sense – would likely be stored in the same WibiData cell.

That’s about all Odiago is disclosing about WibiData right now. Christophe will also be talking at Hadoop World next week, and presumably can be hit up with any burning questions then.

The explanation was led by Oliver Ratzesberger, late of eBay*and progenitor of eBay’s Singularity project. In simplest terms, one event can spawn a lot of event attribute information, perhaps in the form of name-value pairs, which it then makes sense to store together in some way. The example Oliver dwelled on was that, on any given web page, there can be 100+ pieces of information to record, including:

• All 50 search results you were shown, and their positions in the search rankings.
• Every ad, image, or graphical element.
• An ID as to which test you were participating in (every page you see on eBay has some element being tested).

*Oliver is leaving eBay for a still-secret large company. I would conjecture that Michael McIntire is on the move too, either to replace Oliver or to go with him, but Oliver did a very good job of not commenting on the matter.

There are several reasons why one might wish to store this information in ways that grieve relational purists. First, reconstructing all this information via joins would be brutally expensive. What’s more, reconstructing all this information via joins could be impractical. Some comes from third party ad servers, which might not reproduce the same ads upon demand. Other is in the form of rankings, which can’t always be reliably reproduced from one query to the next. (That’s just one of several reasons text search and relational DBMS are an awkward fit.)

Also, there’s a strong dynamic schema flavor to these databases. The list of attributes for one web click might be very different in kind from the list for the next page. Forcing that kind of variability into a fixed relational schema, while theoretically possible, doesn’t necessarily make a lot of sense.

• A long-ago marketing decision to not give infrastructure details, so as to convey a “Don’t worry; we’ll take care of everything” message.
• Even so, a long-ago and perhaps now-regretted marketing decision to disclose and even exaggerate salesforce.com’s reliance on Oracle, as part of an early-days attempt to prove salesforce was using enterprise-class technology.
• A desire to hide the recipe for salesforce.com’s secret sauce.
• Force of habit — I’m not sure salesforce even knows how to tell its technical story with any clarity.

Actually, salesforce.com has moved some kinds of data out of Oracle that previously used to be stored there. Besides Oracle, salesforce uses at least a file system and a RAM-based data store about which I have no details. Even so, much of salesforce.com’s data is stored in Oracle — a single instance of Oracle, which it believes may be the largest instance of Oracle in the world.

Salesforce did spell out some of its database story in a 2008 force.com white paper, which is good stuff, but potentially misleading in one important way. The paper tells of a level of abstraction, whereby what the application sees as logical “columns” are stored in a very different schema than one might assume. However, it doesn’t spell out a second level of abstraction, whereby that logical schema also isn’t how the database is actually laid out.

Another flaw in the paper is that it spins “We had to do this, to support multitenancy, so we did.” issues as “Because we’re multitenant, we can do this, while single-tenant systems can’t.” One example is the query optimization step around “user visibility” in Figure 11. Welcome to marketing.

At the first level of abstraction, data seems to be kept mainly in a single wide table, with hundreds of columns. What’s more, many of those are “flex columns”; a flex column can hold data of many different kinds and even datatypes. Notwithstanding the second level of abstraction, I imagine the idea of stuffing different kinds of thing into the same column has something to do with the fact that Oracle’s physical limit on columns falls far short of the number of logical columns salesforce wants to use.

If we imagine that the different kinds of data in a flex column were each in their own column instead, the whole thing might sound like BigTable/Cassandra/HBase-style column-group NoSQL. Thus, much as Workday uses MySQL to simulate a key-value store, salesforce.com can be said to use Oracle to simulate a different kind of NoSQL. In both cases, what’s going on seems to be a kind of object/relational mapping, but with the relational aspect strongly deemphasized. Or, if you take a more relational view, we could say that salesforce.com’s tables are a lot wider than any one user organization’s, because each user sees only its own custom columns (plus the standard ones common to all users).

The second layer of abstraction has a lot to do with multitenancy. If you want to stick data for many different user organizations into the same huge table, then you have to label it in some way to show who is permitted to see or update each part. Logically, this leads to a join, between one table carrying data plus a simple key showing which users/roles are entitled to see it, and a second table showing who actually is that kind of user/has that kind of role. But that join makes a lot of sense to store in a denormalized way, all the more because data is partitioned across the computer cluster in line with which user organization it actually belongs to.

Multitenant security isn’t the only reason for this denormalization, but it appears to be the biggest one.

The whole thing is doing 550 million or so transactions per day. salesforce.com thinks that fact should be regarded as evidence that it works.

… typical data warehouse appliances, even if they are petascale and parallel, [are] NOT big data solutions.

Nonsense almost as bad can be found in other venues.

Forrester seems to claim that “big data” is characterized by Volume, Velocity, Variety, and Variability. Others, less alliteratively-inclined, might put Complexity in the mix. So far, so good; after all, much of what people call “big data” is collections of disparate data streams, all collected somewhere in a big bit bucket. But when people start defining “big data” to include Variety and/or Variability, they’ve gone too far.

Up to that point, Hopkins — while wrong — is far from alone. The less common part of his error is to further claim that for data to be “big”, it must be stored in a way that violates the C in the CAP Theorem. Yes, the bigger the data set, the more likely that each datum has low individual value, with immediate consistency not being strictly necessary. But there are plenty of big data use cases in which data accuracy turns out to be a good idea.

It actually is reasonable to say that Volume and Velocity of data go together. If you’re storing 5 terabytes of data per day, you have a “big data” kind of problem, whether you then keep it for 30 days or 3000. It also is reasonable to say that Variety and Variability go together; indeed, I’d guess that what Forrester means by those terms corresponds to multi-structured and poly-structured respectively, and using one of those terms is generally plenty.

But while we can whittle four concepts down to two, the reduction should stop there. I say this because any of four combinations is possible (and not just in edge cases):

• Data can be both big and poly-structured. For example, consider the classic Hadoop log-collection use case, or the bigger of MarkLogic’s databases, or of Splunk’s, or even the dynamic-schema parts of relational data warehouses built by Zynga and eBay. And yes, also consider some of the NoSQL-based short-request systems Hopkins was surely thinking of as well.
• Data can be both big and simply-structured. I think most of Teradata’s and Vertica’s petabyte-scale installations would fit that description, the partial counterexamples at eBay and Zynga notwithstanding.
• Data can be not-so-big and poly-structured. Consider, for example, a typical user of Intersystems Cache’.
• Data can be not-so-big and simply-structured. Consider, for example, most of the traditional RDBMS world.

To pretend that those four possibilities are only two — “big data” and otherwise — is a travesty.

If the term “big data” has become useless, then what? Gartner may have switched over to “extreme data,” as reported by my clients at SAND, in honor of the multi-V stuff. That would be an improvement. Better yet would be to stop pretending that a matrix with two dimensions has only one. If what you mean is “huge, poly-structured databases”, then that’s what you should say, or something like it.

If things are bad for “big data”, they’re even worse for “big data analytics”, a term that starts out by inheriting all of big data’s problems and adds more of its own. “Big data analytics” surely means “analytics done on big data” — but nobody’s quite sure what “analytics” are. For example:

• I’m OK with “analytic processing” incorporating all of what might be called business intelligence, visualization (which sometimes now is just the new term for BI), data mining, machine learning, predictive analytics (which for some years has been the term for data mining and machine learning), planning, and yet more. However, …
• … others don’t agree, and contrast “analytics” to “OLAP” and/or to “visualization”, and  seem to equate “analytics” to “predictive analytics” or something similar.
• The latter is what most people have in mind when they say “big data analytics”, but …
• … vendors who can only lay claim to the “analytics” term in its most expansive sense claim to be doing “big data analytics” as well.

Nonsense even worse than Forrester’s ensues.

So here’s what I propose.

• Nobody should ever again say that “big data” doesn’t include big relational data warehouses.
• If your definition of “big data” goes beyond Volume and perhaps Velocity to include Variety, Variability, or Complexity — please call it something else instead. “Extreme data” sounds like a snowboarding competition or something, but at least it’s not as totally erroneous as “big”.
• Never, ever use the phrase “big data analytics” unless you have modifiers near it, to show what kind of big data analytics you’re talking about, or at least to describe the special value you think you bring to the big data analytics process.

Edit: Merv Adrian of Gartner Group has a more reasonable — and wittier! — take than Forrester’s:

You won’t see us telling people “That’s not #bigdata. This is big data.” That’s Crocodile Dundee’s job.

• Derived data.
• Many-step processes to produce derived data.
• Schema evolution.
• Temporary data constructs.

So let’s dive in. 

When I started my discussion of derived data, I focused on five kinds:

Aggregates, when they are maintained, generally for reasons of performance or response time.

Calculated scores, commonly based on data mining/predictive analytics.

Text analytics.

The kinds of ETL (Extract/Transform/Load) Hadoop and other forms of MapReduce are commonly used for.

Adjusted data, especially in scientific contexts.

Later I added a sixth kind:

• Derived metadata, commonly for polystructured data sets (logs, text, images, video, whatever).

More kinds may yet follow.

In all cases, I was (and am) talking about data that is actually persisted into the database. Temporary tables — for example the kind frequently created by Microstrategy — are also important in data processing, as is temp space managed solely for the convenience of the DBMS. But neither are what I mean when I talk about “derived data.”

As I noted back in June, derived data naturally leads to schema evolution. You load data into an analytic database. You do some analysis and get some interesting results — interesting enough for you to want to keep them persistently. So you extend the schema to include them. You do more research; you discover something else interesting; you extend the schema again. Repeat as needed.

However, in no way is derived data the only source of analytic schema evolution. Duh. Sometimes you just have new kinds of information coming in. Of course, once it’s there, you may want to derive something from it. In marketing contexts, both parts of that might be true in spades.

When I mentioned all this to my clients at MarkLogic — which was my inspiration for the polystructured/metadata example — they perked up and said “Oh! Progressive enhancement.” Indeed, it’s long been the case that a simple text processing pipeline could have >15 steps of extraction; indeed, I learned about the “tokenization chain” in 1997. If all the “progression” in  the data enhancement occurs in a single processing run, that wouldn’t necessarily spawn much schema evolution. On the other hand, if you think of additional steps to add every now and then — in that case your schema might indeed evolve over time.

Somewhat similarly, Hadoop can be used to run “aggregation pipelines” of many 10s of steps. The output of the whole thing might be a relatively small number of fields. But again, if the number or nature of the fields changes over time, schemas will need to evolve accordingly.

So to sum up:

• Derived data — of multiple kinds — is very important.
• If you want to increase the value you get from derived data, you might need to evolve your schema accordingly.
• Data derivation happens to sometimes have long processing pipelines; those pipelines might happen to offer clues as how to do yet better at data derivation in the future; those improvements might happen to lead to schema evolution over time.

“Just the raw facts” analytic databases are, for the most part, obsolete.

It’s blindingly obvious that Zynga is one of Vertica’s petabyte-scale customers, given that Zynga sends 5 TB/day of data into Vertica, and keeps that data for about a year. (Zynga may retain even more data going forward; in particular, Zynga regrets ever having thrown out the first month of data for any game it’s tried to launch.) This is game actions, for the most part, rather than log files; true logs generally go into Splunk.

I don’t know whether the missing data is completely thrown away, or just stashed on inaccessible tapes somewhere.

I found two aspects of the Zynga story particularly interesting. First, those 5 TB/day are going straight into Vertica (from, I presume, memcached/Membase/Couchbase), as Zynga decided that sending the data to some kind of log first was more trouble than it’s worth. Second, there’s Zynga’s approach to analytic database design. Highlights of that include:

• Data is divided into two parts. One part has a  pretty ordinary schema; the other is just stored as a huge list of name-value pairs. (This is much like eBay‘s approach with its Teradata-based Singularity, except that eBay puts the name-value pairs into long character strings.) About half the data is in each part, but I don’t think that’s by deliberate choice.
• Zynga adds data into the real schema when it’s clear it will be needed for a while. This isn’t a matter of query volumes, for the most part; rather, it’s when Zynga’s tests (e.g. of new games?) have determined that the data will keep being collected and used for a while.
• Zynga only adds columns to its analytic  database; it never goes through the more complex process of deleting them.

Just as Zynga is one of Vertica’s flagship accounts, LinkedIn is one of Aster Data’s. Specifically, before leaving LinkedIn for Aster, Jonathan Goldman built LinkedIn’s People You May Know feature in Aster nCluster. This was long ago, and I’m not sure how sophisticated his use of SQL and MapReduce would be in today’s terms; for example, I was told he didn’t use “nPath or anything like that.” (Edit: See the comments below for clarifications from Jonathan.) Anyhow, LinkedIn has replaced Aster for PYMK with Hadoop, and in my opinion is getting much better results.

That, from an Aster standpoint, is the bad news. The good news is that LinkedIn is happily using Aster nCluster for several other applications; LinkedIn folks doesn’t seem to regret throwing out* Greenplum for Aster; and they also seem to have a very high opinion of Jonathan and his work while he was there.

*And this time that is indeed the phrase that was used.

One thing that astonished me is that LinkedIn PYMK is based only on data innate to LinkedIn (as opposed to imported email addresses, the results of web crawls, and so on). Given that, I am at a loss to explain how it suggested a couple of old friends, to whom I have no discernable chain of connection. Yes, we were at Harvard at the same time, but if that’s all it was, there would be a huge number of false positives I’m not actually seeing.

• “A bunch of apps in one, similar but not the same” (in my recent post on MongoDB).
• Out-of-control product catalogs (ditto).
• Analytic use cases in which one keeps enhancing the database with derived data.

And if marketing profile analysis is ever done correctly, that will be a huge example for the list.

So what do we call those DBMS — for example NoSQL, object-oriented, or XML-based systems — that bake the schema into the applications or the records themselves? In the MongoDB post I went with “schemaless,” but I wasn’t really comfortable with that, so I took the discussion to Twitter. Comments from Vlad Didenko (in particular), Ryan Prociuk, Merv Adrian, and Roland Bouman favored the idea that schemas in such systems are changeable or late-bound, rather than entirely absent. I quickly agreed.

The discussion wasn’t entirely serious; wise-ass comments were contributed by at least Merv, Neil Raden, Yiorgos Adamopoulos, and myself.

I like that approach for the same reason I favor saying that databases are poly- or multi-structured (rather than un- or semi-):  Every database has structure, the only question being when that structure is determined. I wouldn’t precisely equate “poly-structured” to “has a late-bound schema”; for example, I’d say that mucking with the DDL (Data Description Language) of a relational database shows that it’s a little bit poly-structured, even though it’s not at all late-bound. But the concepts are definitely related.

So what actual wording should we use here? The only alternative I see to fixed schema is “static”, and that feels like it has too much of a connotation of “unchangeable”. The simplest word I can think of for changeable/late-bound/whatever is dynamic schema; that choice also has the virtue of some traction, as per the Vlad Didenko tweet linked above. Casual googling is also supportive of “fixed” and “dynamic”, at least over whatever alternatives I came up with. So those are my choices.

For actual definitions, I’ll say:

• A (logical) schema is fixed if it is defined before a program is written, but dynamic if it is defined by the program or data itself.
• A database is fixed- or dynamic-schema depending on whether its schemas are fixed or dynamic respectively.
• A DBMS is fixed- or dynamic-schema depending on whether databases created in it tend to have fixed or dynamic schemas respectively.

Suit yourself as to what you say about relational DBMS when they also have a bit of XML, text, or whatever support.

By these definitions:

• Relational databases are fixed-schema (within the caveat above about XML or text data).
• MOLAP databases are fixed-schema.
• Pre-relational network and hierarchical DBMS (e.g. IMS) are fixed-schema.
• Most other DBMS are dynamic-schema.

What do you think? Do these definitions work for you?