Interesting quotes include:

WibiData is designed to store … transactional data side-by-side with profile and other derived data attributes.

… the ability to add new ad-hoc columns to a table enables more flexible analysis: output data that is the result of one analytic pipeline is stored adjacent to its input data, meaning that you can easily use this as input to second- or third-order derived data as well.

schemas can vary over time; you can easily add a field to a record, or delete a field. … But even though you start collecting that new data, your existing analysis pipelines can treat records like they always did; programs that don’t yet know about the new cookie are still compatible with both the old records already collected, and the new records with the additional field. New programs fill in default values for old data recorded before a field was added, applying the new schema at read time.

schemas for every column are stored in a data dictionary that matches column names with their schemas, as well as human-readable descriptions of the data.

Interesting aspects of the post that don’t lend themselves as well to being excerpted include:

• How the Produce-Gather “analysis calculus” — i.e. framework — works.
• How this all ties into Apache projects (and sub-projects) such as Hadoop, HBase, and Avro.

• Sumo Logic does SaaS (Software as a Service) log management.
• Sumo Logic is text indexing/Lucene-based. Thus, it is reasonable to think of Sumo Logic as “Splunk-like”. (However, Sumo Logic seems to have a stricter security/trouble-shooting orientation than Splunk, which is trying to branch out.)
• Sumo Logic has hacked Lucene for faster indexing, and says 10-30 second latencies are typical.
• Sumo Logic’s main differentiation is automated classification of events.
• There’s some kind of streaming engine in the mix, to update counters and drive alerts.
• Sumo Logic has around 30 “customers,” free (mainly) or paying (around 5) as the case may be.
• A truly typical Sumo Logic customer has single to low double digits of gigabytes of log data per day. However, Sumo Logic seems highly confident in its ability to handle a terabyte per customer per day, give or take a factor of 2.
• When I asked about the implications of shipping that much data to a remote data center, Sumo Logic observed that log data compresses really well.
• Sumo Logic recently raised a bunch of venture capital.
• Sumo Logic’s founders are out of ArcSight, a log management company HP paid a bunch of money for.
• Sumo Logic coined a marketing term “LogReduce”, but it has nothing to do with “MapReduce”. Sumo Logic seems to find this amusing.

What interests me about Sumo Logic is that automated classification story. I thought I heard Sumo Logic say:

• It’s largely unsupervised machine learning.
• It’s specific to a particular user/data set.
• It can be up and running and classifying things effectively almost instantly (i.e., on seconds’ or minutes’ worth of data).
• It’s informed by what different users tag as false positives. (Or maybe that is planned for future versions.)

I have a little trouble seeing how all those points fit exactly together, so perhaps I got some details wrong.

The payoff is that machine learning directly informs the Sumo Logic user interface. In particular, large numbers of events are bundled into a small number of categories, hopefully making it much easier for network operations types to scan the UI and pick out what’s important.

In general, the idea of machine-learning informing analytic UIs via some sort of classification is common in text-oriented technologies, notably in:

• Good ol’ text search.
• Text mining vendors’ approaches to clustering hits on words or phrases that say substantially the same thing.

But otherwise it seems kind of rare, if we stipulate that ad-serving/general internet personalization isn’t really an analytic UI — but I’d love to hear of any interesting examples I’ve overlooked.

• The Forrester Wave’s relative vendor rankings are meaningless, in that the document compares apples, peaches, almonds, and peanuts. Apparently, it covers any vendor that includes a distribution of Apache Hadoop MapReduce into something it offers, and that offered at least two (not necessarily full production) references for same.
• The Forrester Wave for “enterprise Hadoop” contradicts itself on the subject of Hortonworks.
  • The Forrester Wave for “enterprise Hadoop” is correct when it says “Hortonworks … has Hadoop training and professional services offerings that are still embryonic.”
  • Peculiarly, the Forrester Wave for “enterprise Hadoop” also says “Hortonworks offers an impressive Hadoop professional services portfolio”. Hortonworks will likely win one or more nice partnership deals with vendors in adjacent fields, but even so its professional services capabilities are … well, a good word might be “embryonic”.
• Forrester Waves always seem to have weird implicit definitions of “data warehousing”. This one is no exception.
• Forrester gave top marks in “Functionality” to 11 of 13 “enterprise Hadoop” vendors. This seems odd.
• I don’t know why MapR, which doesn’t like HDFS (Hadoop Distributed File System), got top marks in “Subproject integration”.
• Forrester gave top marks in “Storage” to Datameer. It also gave higher marks to MapR than to EMC Greenplum, even though EMC Greenplum’s technology is a superset of MapR’s. Very strange. (Edit: Actually, as per a comment below, there is some uncertainty about the EMC/MapR relationship.)
• Forrester gave higher marks in “Acceleration and optimization” to Hortonworks than to Cloudera and IBM, and higher marks yet to Pentaho. Very odd.
• I’m not sure what Forrester is calling a “Distributed EDW file store connector”, but it sounds like something that Cloudera has provided via partnership to a number of analytic DBMS vendors.
• Forrester’s “Strategy” rankings seem to correlate to a metric of “We’re a large enough vendor to go in N directions at once”, for various values of N.
• Forrester is correct to rank Cloudera’s “Adoption” as being stronger than EMC/Greenplum’s or MapR’s. But Hortonworks’ strong mark for “Adoption” baffles me.

Three things that absolutely should NOT be obstacles to these ends are:

• Corporate DBMS standards.
• Corporate data governance processes.
• The difficulties of ETL.

Reasons they shouldn’t or don’t need to be obstacles include:

• Analytic DBMS are often vastly more cost-effective than general-purpose ones.
• In particular, analytic DBMS are often much easier to install and manage than general-purpose ones.
• Heavy data governance bureaucracy is often unnecessary because:
  • The department should know what the limitations on the data’s accuracy are.
  • The department should know how much data accuracy is required.
  • The side-effects on other departments of any data inaccuracy would be minimal.
• There are multiple good schemes for populating data marts, managed by cost-effective analytic DBMS, with data from integrated data warehouses.
  • ELT (Extract/Load/Transform) almost always works, because data cleaning/data quality was handled at or before the IDW level, and because the analytic DBMS has the processing power to pull it off.
  • ETL (Extract/Transform/Load) should be easy as well. (If isn’t, something may be lacking in your ETL set-up.)
  • Analytic DBMS are increasingly adding capabilities for easy spin-out of real or virtual data marts. Other kinds of technology (e.g. virtualization) are having their database spin-out capabilities upgraded as well.

One point to remember in support of departmental autonomy is that departments’ views of what data to use may be more expansive than central IT’s. One reason is that important data may be external to the company, outside IT’s natural realm  of concern. Examples of this include but are hardly limited to:

• Anything like “market data”.
• Anything like “sentiment analysis”.
• Data owned by supply chain partners.

Further, even the more innovative internal data sources are commonly departmental, for example various kinds of multi-structured data (text verbatims from customers, log file data, and so on).

Whatever is true of data management (and ETL) is true for metadata management, even if it’s done by some kind of business intelligence tool. What I mean by that is:

• Whoever manages data is also responsible for ingesting and emitting it …
• … and specifically for emitting it in understandable, well-organized, well-named formats, …
• … so that departments can take responsibility for what amounts to lightweight analytic application development.

As for the “application development” itself, I’m envisioning at least three things:

• Math.
• Sophisticated relational query.
• Data visualization.

I.e., I’m talking about what “analysts” and “quants” do. So to put the point even more simply:

• Analysts and quants should be able to consume data that’s organized in a friendly manner.
• Central IT should be friendly in how it serves data.

One corollary of this approach is that departments should try to adhere to corporate BI standards, at least for routine dashboard and reporting. Indeed, if a department brings in a business intelligence tool different from the corporate standard, there are three main possibilities:

• The tool is integrated with something else it makes sense to bring in, such as a third-party data supply or application.
• The tool has an important capability the corporate standard doesn’t have, such as more flexible visualization and drilldown.
• Central IT screwed up, making things much more difficult than they needed to be.

Reporter: [Care to comment]?
CAM: SQL Server is an adequate product if you don’t mind being locked into the Microsoft stack. For example, the ColumnStore feature is very partial, given that it can’t be updated; but Oracle doesn’t have columnar storage at all.

Reporter: Is the lock-in overall worse than IBM DB2, Oracle?
CAM: Microsoft locks you into an operating system, so yes.

Reporter: Is this release something larger Oracle or IBM shops could consider as a lower-cost alternative a co-habitation scenario, in the event they’re mulling whether to buy more Oracle or IBM licenses?
CAM: If they have a strong Microsoft-stack investment already, sure. Otherwise, why?

Reporter: [How about] just cost?
CAM: DB2 works just as well to keep Oracle honest as SQL Server does, and without a major operating system commitment. For analytic databases you want an analytic DBMS or appliance anyway.

Best is to have one major vendor of OTLP/general-purpose DBMS, a web DBMS, a DBMS for disposable projects (that may be the same as one of the first two), plus however many different analytic data stores you need to get the job done.

By “web DBMS” I mean MySQL, NewSQL, or NoSQL. Actually, you might need more than one product in that area.

• Many analytic challenges are inherently departmental.
• In many cases, central IT control of analytics isn’t needed.
• Departments move ahead without central approval or involvement because they can.

That said, arguments for centralizing analytic technology include:

• A lot of data is used by more than one department, for example:
  • Financial transactions (one or more affected departments and also the central accounting group).
  • Web logs (marketing and IT/web operations).
• Departments may not have the requisite technical expertise (and it may be redundant/cost-ineffective for them to acquire it).

What’s more, there are IT best practices to support department-level analytics. Some of the key ones boil down to:

• Be flexible in your analytic DBMS support.
• Be responsive to requests for ETL.

My conclusion is that central IT should encourage (and aid) departmental analytics. Let’s look at some details.

I think two huge categories of analytic problem are inherently departmental:

• Investigative analytics (pretty much all of it).
• Routine monitoring/dashboarding if the data is tracked just by one department.

Investigative analytics is a kind of research activity — you’re looking to discover previously unrecognized patterns. There are two approaches to this — you can do it in the department that has the relevant business knowledge, or you can outsource it to a special group of “discoverers” (commonly statisticians).* Either way, this is a small team/departmental kind of activity.

*Combining the two approaches is common — a department can have its own analytically adept discoverers, whether they’re call “quants” or just “business analysts”.

Reporting/monitoring BI at least has the potential to be enterprise-wide — but commonly it isn’t, as each department has its own operational data sources and metrics. Marketing departments may watch external data that the rest of the company doesn’t worry about. But it can be true across the board. Factory operations folks may track machine tool data the rest of us barely understand.

Even if a business need is strictly departmental, there can be at least two reasons to centralize technology implementation:

• The department doesn’t have the critical mass of IT expertise.
• Departmental IT has side effects on the rest of the company.

Whether those reasons hold up depends a lot on what kind of analytic scenario we’re talking about.

Let’s organize that part of this discussion in line with the taxonomy from my eight kinds of analytic database posts last July.

• Enterprise data warehouses fall under the purview of major IT organizations. That remains true even if we pivot to the more realistic concept of integrated data warehouse. However, less stuff needs to be protected in an EDW/IDW than some data authoritarians like to think.
• I wrote that the stresses on traditional data marts were “performance, concurrency, TCO.” This is a clue that the more demanding examples are right in IT’s wheelhouse. As for the less demanding cases — IT should be able to meet those needs without breaking a sweat.
• Agile investigative data marts are inherently departmental. If you have the talent to use one, you also have the talent to, for example, train into being a part time Netezza DBA. Who cares if you don’t have the expertise to do sophisticated tuning? Analytic DBMS are fast enough — and hardware is cheap enough — that you don’t that skill set anyway.
• Big investigative data marts can go either way. They’re technically challenging, so IT certainly has a claim on them. But in cases where the data, while big, is fairly homogeneous, it’s also not unrealistic for departments to handle the mart themselves.
• Bit buckets are often departmental today, with the department in question happening to be central IT. And central IT is where they’re likely to flourish, as the data they hold becomes ever more diverse.
• Archival data stores are a central IT matter. Nobody else is likely to care enough to do it right.
• Outsourced data marts, by definition, don’t live inside conventional enterprises. But they are often a way for business units to get access to data and analytics without relying on central IT.
• Operational analytics servers are likely to be sufficiently mission-critical that you want them handled by IT.

So in most cases I’d say: Departments can manage their own investigative data marts, and so of course can SaaS vendors and third-party data providers; other analytic databases should be run by central IT. (And of course, large departments with serious local IT can fuzz those distinctions up.) Beyond that, it would seem that whoever administers the database should administer the rest of the analytic stack as well.

That still leaves us with some practical questions, such as:

• Exactly what products should IT departments buy for which purposes? I hope a lot of posts in this blog are helpful in that consideration.
• How should development tasks be split between departments and central IT? It may take me a while to get a post together on that the subject, since in general the analytics-development picture is pretty complicated to lay out.
• How should departments and central IT work together to manage departmental investigative data marts? I hope to post on that subject soon.

Excerpts from the first post include (with minor edits for formatting, including added emphasis):

Back in 1995, Vladimir Vapnik … changed the machine learning game with his new ‘Statistical Learning Theory’: he provided the machine learning guys with a mathematical framework that allowed them finally to understand, at the core, why some techniques were working and some others were not. All of a sudden, a new realm of algorithms could be written that would use mathematical equations instead of engineering data science tricks (don’t get me wrong here: I am an engineer at heart and I know the value of “tricks,” but tricks cannot overcome the drawbacks of a bad mathematical framework). Here was a foundation for automated data mining techniques that would perform as well as the best data scientists deploying these tricks. Luck is not enough though; it was because we knew a lot about statistics and machine learning that we were able to decipher the nuggets of gold in Vladimir’s theory.

The market needed a system that is able to perform classification and regression (we later added clustering/segmentation, times series analysis, association rules and social network analysis), that has the following characteristics:

1. Non-parametric: little user intervention and tuning should be required — it should work well out of the box.
2. Independent of the data and target distribution:
   • Target: the classification system should be able to handle rates of positive values even as low as 0.1% (such as in fraud, for example), or be able to forecast a continuous value with only 1% of non zero values.
   • Data: it should automate mixing and matching and comparing influence for ordinal, nominal, continuous,and textual variables without any user intervention.
3. Scalable in number of rows: the training time should be linear with the number of rows, and the quality of the models should increase with the number of rows.
4. Scalable in number of columns: the training time should be close to linear with respect to the number of columns, and the quality of the models should increase with the number of columns. It is well known that most algorithms present a problem of over-fitting in high dimensions; it is quite ironic that companies spend billions of dollars in collecting data but often cannot take advantage of all this data because most first-generation analytical workbenches collapse trying to handle the high dimensionality inherent in all this data.
5. Descriptive: a good predictive analytics package must be able to present its findings in a way that a business user can understand. We have always believed that there is a continuum between predictive and descriptive analytics: predictive models should be descriptive enough and descriptive models should be usable in a predictive manner to make decisions.
6. Deployable: the scoring equations should be simple enough to be deployed in any operational environment: SQL for databases, Java code for the web (or even for smartphones), etc.

Vapnik’s theory provided us with a mathematical framework for capabilities 1, 2 and 4 above; what remained was 3, 5 and 6, which we solved with a well known pattern in machine learning: by using linear systems in a properly encoded space (the trick is to have the good encoded space).

The second post seems to make some strong model-quality benchmark claims, but there also seems to be an in-house-vs.-publicly-checked mismatch going on.

• Splunk is first and foremost an analytic DBMS …
• … used to manage logs and similar multistructured data.
• Splunk’s DML (Data Manipulation Language) is based on text search, not on SQL.
• Splunk has extended its DML in natural ways (e.g., you can use it to do calculations and even some statistics).
• Splunk bundles some (very) basic, Splunk-specific business intelligence capabilities.
• The paradigmatic use of Splunk is to monitor IT operations in real time. However:
  • There also are plenty of non-real-time uses for Splunk.
  • Splunk is proudest of its growth in non-IT quasi-real-time uses, such as the marketing side of web operations.

As in any release, a lot of Splunk 4.3 is about “Oh, you didn’t have that before?” features and Bottleneck Whack-A-Mole performance speed-up. One performance enhancement is Bloom filters, which are a very hot topic these days. More important is a switch from Flash to HTML5, so as to accommodate mobile devices with less server-side rendering. Splunk reports that its users — especially the non-IT ones — really want to get Splunk information on the tablet devices. While this somewhat contradicts what I wrote a few days ago pooh-poohing mobile BI, let me hasten to point out:

• Splunk is used for a lot of (quasi) real-time monitoring.
• Splunk’s desktop user interfaces are, by BI standards, quite primitive.

That’s pretty much the ideal scenario for mobile BI: Timeliness matters and prettiness doesn’t.

Hmm. Maybe StreamBase LiveView needs a mobile option as well …

Splunk’s basic use is to take the text string that is a log and make sense of it. But Splunk now also supports JSON structures. It does this via something called spath, which as you might guess from the name has XPath similarities. That probably bore more discussion than we found the time to have.

By the way: If you’re interested in BI over XML, that’s what my former clients at Skytide were founded to do, before they pivoted a bit. I don’t think those capabilities have disappeared from the product.

Splunk has graciously allowed me to post a slide deck. More stuff in there, including quotes from a customer — Expedia — that has 2700 Splunk users.

• Bigness — Volume, Velocity, size
• Structure — Variety, Variability, Complexity

given that

• High-velocity “big data” problems are usually high-volume as well.*
• Variety, variability, and complexity all relate to the simply-structured/poly-structured distinction.

But the conflation should stop there.

*Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

When people claim that bigness and structure are the same issue, they oversimplify into mush. So I think we need four pieces of terminology, reflective of a 2×2 matrix of possibilities. For want of better alternatives, my suggestions are:

• Relational big data is data of high volume that fits well into a relational DBMS.
• Multi-structured big data is data of high volume that doesn’t fit well into a relational DBMS. Alternative: Poly-structured big data.
• Conventional relational data is data of not-so-high volume that fits well into a relational DBMS. Alternatives: Ordinary/normal/smaller relational data.
• Smaller poly-structured data is data for which dynamic schema capabilities are important, but which doesn’t rise to “big data” volume.

Notes on all this include:

• “Relational big data” is commonly what you need a scalable analytic relational DBMS for. But there are non-analytic use cases as well.
• The paradigmatic example of “multi-structured big data” is log files. Thus, multi-structured big data is commonly what you need a big bit bucket for.
• One might want to equate non-analytic relational big data technology to “NewSQL”. However, I’m struggling to think of a database size range in which the entire NewSQL industry can match Oracle’s market share alone.
• One might want to equate non-analytic multi-structured big data technology to “NoSQL”. However:
  • “NoSQL” is also used to encompass not-so-big-data use cases, such as prototyping in MongoDB.
  • “NoSQL” has non-ACID/low(er)-data-integrity connotations that aren’t appropriate for all non-relational systems.
• Up to a point, you can analyze relational big data in a conventional relational DBMS, but an analytic RDBMS will usually win on TCO (Total Cost of Ownership). In particular, reasonable thresholds for moving an analytic database off Oracle might be:
  • 1-2 terabytes if you’ve never bought anything past Oracle Standard Edition.
  • 5-10 terabytes if you’re already paying for Oracle Enterprise Edition.
  • A lot higher than that if you actually find Oracle Exadata to be cost-effective.
• Depending on how big one acknowledges as “big”, the market share leader in “big bit bucket” use cases is either Splunk or Hadoop.
• If we look at multi-structured big data management overall, MarkLogic joins the list of market share contenders, as do various NoSQL alternatives.
• It is wrong to say that the large web companies invented “big data” technology. But it is more reasonable to say they invented much of “multi-structured big data” management. In particular (and this is just a partial list), Google, Amazon, Yahoo, Facebook, et al. can reasonably be credited with Hadoop, Cassandra, HBase and various predecessors to same.

Mobile business intelligence — real business value or just a snazzy demo?

I discussed some mobile BI use cases in July 2010, but I’m still not convinced the whole area is a legitimate big deal. BI has a long history of snazzy, senior-exec-pleasing demos that have little to do with substantive business value. For now, I think mobile BI is another of those; few people will gain deep analytic insights staring into their iPhones. I don’t see anything coming that’s going to change the situation soon.

BI-centric collaboration — real business value or just a snazzy demo?

I’m more optimistic about collaborative business intelligence. QlikView’s direct sharing of dashboards will, I think, be a feature competitors must and will imitate. Social media BI collaboration is still in the “mainly a demo” phase, but I think it meets a broader and deeper need than does mobile BI. Over the next few years, I expect numerous enterprises to establish strong cultures of analytic chatter (and then give frequent talks about same at industry conferences).  

Business intelligence for mid-market enterprises is problematic

Given the saturation of the large-enterprise BI market with supposed enterprise-standard BI systems, it would seem that smaller enterprises comprise a large part of the BI growth opportunity. However, the large-enterprise and mid-range BI markets are very different. For example:

• Large enterprises typically have tough challenges in data integration; smaller enterprises may truly start out with their data in only a few systems.
• There are many reasons for large enterprises not to do their BI in the cloud, such as bandwidth, internal politics, or the unsuitability of most cloud infrastructure for analytic DBMS scale-out. Smaller enterprises, however, may prefer SaaS (Software as a Service) BI.
• The BI market for smaller enterprises is heavily OEM. But unless you’re buying some kind of data/analytics bundle, the large enterprise BI market still seems overwhelmingly standalone.
• Large-enterprise BI tools incorporate much of a DBMS-like technology stack; at smaller enterprises, BI can often stick to its specialized-application-development-tool knitting. But on the other hand …
• … large enterprises almost always already have a data warehousing infrastructure. Mid-range BI buyers may not have a separate analytic DBMS. Therefore …
• … BI/DBMS bundles make more sense in the mid-market than they do at large enterprises.
• Each large enterprise has a unique infrastructure, and  commonly a unique competitive situation as well. Thus, the idea that you’ll pre-build most of an analytic application for a large enterprises — because you know what data model they need to do their BI — usually turns out to be silly. But smaller enterprises can be more homogeneous, and so for them pre-built analytic applications can actually work.

I don’t know of anybody who’s really cracked the code on mid-market BI. Crystal Reports (long owned by SAP Business Objects) has huge OEM share, but somehow hasn’t parlayed that into a comprehensive mid-market BI presence. Various SaaS or on-premise vendors have cool product ideas — e.g. Gooddata, Endeca, or my clients at PivotLink — but none seems to have set the world on fire to this point.

Departmental BI is doing better

The news is happier in a related market — business intelligence for departments of larger enterprises. However, this is a hard market to analyze, for at least two reasons. First — as is often the case — the distinction among large-enterprise-wise, smaller-enterprise-wide, and departmental BI is not a clear one.* Second, “departmental BI” has at least two major strains:

• Simple, pedestrian BI, implemented quickly.
• Investigative analytics.

*In particular, it has been the case since the 1990s that BI tools first get sold to departments, hopefully for fast implementations — think 4-6 weeks as a base case — and then spread out internally after their initial successes. I am frequently amused by vendors who claim to have pioneered that sales model sometime over the past decade, or even within the past few years.

That said, there are two main kinds of reason to do your BI departmentally, at arm’s length from central IT.

• Perhaps, for good reason or bad, IT is being insufficiently helpful at managing the data.
  • This can be a straightforward matter of politics and priorities — IT controls the data, but is slow about giving you access.
  • Also, you may want to include data that’s outside IT’s purview, be it third-party or just purely departmental.
• Further, you may want functionality that corporate-standard BI doesn’t offer. Potential examples include:
  • Cool analytic visualization.
  • “Real-time” data visualization.
  • The ability to play nicely with particular kinds of data sets.

I have a lot more to say about those points — but not in a post that’s already as long as this one.