I mean for the term “data mustering” to encompass at least three scenarios:

• Integrated (relational) data warehouse.
• Big bit bucket.
• Big bit stream.

Let me explain what I mean by each. 

“Integrated data warehouse” is a phrase Teradata has started using for enterprise data warehouses that, like approximately every other EDW in the entire history of data warehousing, aren’t truly enterprise-wide. In other words, it means “not just a data mart”. No category name is perfect, but I think that one works reasonably well.

I previously described the big bit bucket use case as

Users take a whole lot of data, often machine-generated data in logs of different kinds, and dump it into one place, managed by Hadoop, at open-source pricing.

and quickly added

Of course, there are various outfits who’d like to sell you not-so-cheap bit buckets. Contending technologies include Hadoop appliances (which I don’t believe in), Splunk (which in many use cases I do), and MarkLogic (ditto, but often the cases are different from Splunk’s). Cloudera and IBM, among other vendors, would also like to sell you some proprietary software to go with your standard Apache Hadoop code.

I think I’ll stand pat on that explanation.

By analogy, a big bit stream is various streams of data, assembled in the custody of a streaming engine. Sybase told me Wednesday that this scenario appears in both of the traditional markets for CEP/streaming — national intelligence, where it is a major use of streaming, and capital markets in some use cases as well. And it’s consistent with what I’ve heard from other CEP/streaming vendors as well.

As for where I got the word “mustering” — it’s a military term, for when you assemble your troops and their gear either for inspection or for actual use. The main modern usage I know of the word is as part of the phrase “pass muster”, which originally referred to the concept that the person being paid to put a regiment together should from time to time demonstrate that the regiment physically existed in the form that regimental records seemed to show.

• LiveView is a business intelligence and alerting suite built on/in the rest of StreamBase’s technology, meant to operate on streaming data.
• LiveView is positioned by StreamBase as having a true push event-driven architecture rather than pull/poll.
• StreamBase LiveView is designed to query in-memory data and then have the results change in real time as the data set changes.
• The LiveView user interface is a rapidly changing work in progress.
• LiveView has other Version 1 limitations as well
• LiveView is targeted squarely at StreamBase’s financial trading core market until some of the Version 1 limitations are lifted.

The basic StreamBase LiveView pipeline goes something like:  

• Data comes into the system via multiple streams.
• Transformations upon data arrival can include but are not limited to:
  • Aggregations.
  • Joins to reference data.
  • Joins to other streams.
• The streams (transformed or perhaps otherwise) are output to tables …
• … which are continuously updated as more data streams through.
• The data in the resulting table can be consumed:
  • Via LiveView-provided BI capabilities.
  • Via an API.

When wearing my vendor consultant hat, I warmly encourage StreamBase to emphasize the lack of a batch step anywhere in this process. As an analyst, however, I’m more restrained about a claim like “We uniquely free you from batch.” I agree that avoiding batch jobs is a Very Nice Thing. But you also are spared most batch-cycle processing if you stream updates from your short-request database to an analytic DBMS, e.g. via some kind of near-real-time replication.

That said, the push-versus-pull continuous filtering part of the StreamBase LiveView story seems pretty real. I think having sub-second display updates is cool in all sorts of BI use cases, and seriously useful in some number of them. While I don’t have a clear opinion as to whether the StreamBase approach offers huge performance advantages for that kind of latency over “pull” alternatives, my guess is in the direction of “yes”.

Version 1 limitations on StreamBase LiveView include:

• You consume data one table at a time, with no possibility of a join after the data has originally been put into a LiveView table.
• While LiveView in principle offers rich alerting potential, you get at it via an API rather than much in the way of alerting-specific tools.
• The first LiveView UI StreamBase put together looks a lot like 1980s stock quote machines. The next one it added looks a lot like Panopticon. Much cool-looking enhancement remains to be done.

One competitive (non)-note: This all sounds something like what TIBCO has been pushing for years, but in fact I don’t have much knowledge of TIBCO’s efforts in the area. I had a meeting set up to learn about it some time ago, but it got canceled because TIBCO’s PR people:

• Didn’t want to let any kind of meeting happen without them, even though a serious CTO-type representative seemed happy to talk, but also …
• … didn’t want to work at dinner time.

I haven’t had substantive contact with TIBCO since.

• StreamBase once planned to conquer the world.
• However, StreamBase really only sold effectively in the financial trading and intelligence markets.
• StreamBase retrenched, focusing almost exclusively on the financial trading market.
• With StreamBase LiveView, StreamBase is expanding from embedded operational analytics to do (also operational) business intelligence as well.
• StreamBase is hopeful that, perhaps starting with Version 2 or so, LiveView will be successful outside the financial trading market.

Not coincidental to these shifts in focus, StreamBase was our client, then stopped being one for a while, and now is a client again.

StreamBase (the product set) consists primarily of three things (LiveView aside):

• A development environment, whose output is in …
• … a visual programming language called EventFlow …
• … which is complied and executed by StreamBase’s execution layers.

One important set of ancillary products are StreamBase’s connectors to various data sources — StreamBase offers about 125 of its own, a number that approaches 200 when community contributions are included.

StreamBase has a second programming language called StreamSQL, but that’s rarely used except for embedding in or connecting to third-party software. EventFlow and StreamSQL compile to nearly identical byte code. (The main difference seems to be that as a practical matter you’ll name things a bit differently in the two languages, focusing on verbs in EventFlow and nouns in StreamSQL.)

StreamBase says that in the financial trading market, great performance out of the box equates to better time-to-value, since you are spared time you’d otherwise have to spend tuning the system. Implicit in that is a claim — which competitors might dispute — that StreamBase has great performance. StreamBase fondly thinks that having a domain-specific language gives it a leg up in achieving great compiler optimization. (The same would presumably apply to StreamBase’s competitors, but only if they have optimizing compilers themselves.)

One point that’s a little unusual for me these days is that StreamBase favors big SMP (Symmetric MultiProcessing) boxes over blade-based scale-out. 16+ cores and 256 gigabytes of RAM are not uncommon. Clusters commonly include 4-8 machines, but rarely more; the largest StreamBase cluster evidently contains 36 machines.

And with that I’ll turn to StreamBase’s newest offering, LiveView.

• The upshot of my what to call CEP thread in August was that “streaming” and “event processing” are not the same concept, but it so happens that they have the most traction where they intersect. That said, I both observe and endorse an apparent shift from “event” to “stream” as the core of the terminology, in a reversal of my opinion of several years ago.
• IBM continues to throw a lot of resources at its System S/ InfoSphere Streams product, but I haven’t heard yet of much marketplace success. That said, I believe IBM is still pretty serious about Streams, as one would expect from an effort whose code name so cheekily references System R. In particular, Streams shows up prominently on IBM’s top-level analytic architecture slide.
• Sybase recently released its ESP (Event Stream Processor) 5.0, which it says is the full merger of the Aleri and Coral8 predecessors. You can still get Sybase ESP without buying into the full Sybase RAP stack, and Sybase has no plans to change that.
• Sybase has discontinued all the business intelligence types of products Aleri and Coral8 were developing. Rather, Sybase is OEMing Panopticon, which it reports has been well received. Other than the discontinuation of the BI efforts, there seem to be few Aleri or Coral8 features missing from the merged Sybase ESP product.
• Truviso continues to be out of the picture.
• I have more to say about StreamBase separately.
• I have more to say about Sybase and IBM, which I’ll get to when I can.
• I have nothing new on Progress Apama. I also know little about any of the open source efforts.

Meanwhile, if you want to see technically nitty-gritty posts about the CEP/streaming area, you may want to look at my CEP/streaming coverage circa 2007-9, based on conversations with (among others) Mike Stonebraker, John Bates, and Mark Tsimelzon.

The other main alternative is to base the name on “stream processing” instead.* The CEP-or-whatever industry is split between these choices, with StreamBase currently favoring “CEP” (despite its company name), IBM emphatically favoring “stream”, and Sybase seemingly trying to have things both ways.

*And then, of course, there is “event stream processing”, regarding which please see below.

I’ve been juggling this terminological divide myself, referring to complex event/stream processing as long as four years ago. But enough is enough. I’d like to write more about the category without repeatedly apologizing for its name. And so, always bearing in mind Monash’s Third Law of Commercial Semantics, here’s where I’m coming down.

The more I think about it, the less I like the term “event processing”. Here’s why. Events happen; data is produced; CEP systems most commonly try to identify and categorize the events based on the data. The CEP systems may then do significant further processing, but more often they just pass the information on to another system (most commonly either persistent DBMS or “real-time” business intelligence). How much of that is really “event processing”? Relatively little, I’d say. And referring specifically to “complex” events doesn’t address my complaints at all.

So I’d like to go with some version of “stream”. But “stream processing” has other computer-related uses, while “Stream management” commonly describes care and planning for small waterways. So “stream” might do best with a modifier, such as “event” or “data”. Of the two, I prefer “data stream” (or “datastream”) to “event stream”; the events aren’t really streaming, but the data is.

So should it be “data stream processing” or “data stream management”? Well, the only one of numerous Wikipedia definitions I’ve actually liked while researching this post is the one for “Data Stream Management System“:

A Data Stream Management System (DSMS) is a set of computer programs that controls the maintenance and querying of data in data streams. The use of a DSMS to manage a data stream is roughly analogous to the use of a Database Management System (DBMS) to manage a conventional database.

A key feature of a DSMS is the ability to execute a continuous query against a data stream. A conventional database query executes once and returns a set of results for a given point in time. In contrast, a continuous query continues to execute over time, as new data enters the stream. The results of the continuous query are updated as new data appears.

I think the data stream/database management analogy is spot on. Your queries work a little differently, but otherwise you’re doing pretty much the same things. Indeed, you’re probably even going to persistently store some of the data, and ideally that DBMS capability would be tightly integrated into your CEP system. (In practice they’re apt to be more loosely coupled; for most purposes that works well enough.) Query execution, data ingestion, performance monitoring/tuning, workload prioritization — it’s very DBMS-like stuff. And by the way, “data stream management system” is the term that was used by the researchers — Mike Stonebreaker, Stan Zdonik, Dan Abadi, et al. — who wrote a paper describing the project on which StreamBase was based … although some might question whether that particular observation is a strong signal of accuracy.

This reasoning suggests Data Stream Management System is what it should be. The usual kinds of abbreviation — datastream (product), datastream manager, DSMS, etc. would no doubt follow. So should it be “Data Stream”, “Datastream”, or “Data-stream”? At that level of detail, I don’t yet have an opinion.

The only thing is — that’s all pretty wordy compared to CEP. So after all this, I’m still not sure which term(s) I prefer.

What are your thoughts?

Bit bucket

• Kinds of data likely to be included: Logs, other technical/external
• Likely use styles: Staging/ETL, investigative
• Canonical example: Log files in a Hadoop cluster
• Stresses: TCO, scale-out, transform/big-query performance, ETL functionality

With the explosion of machine-generated data has come the need for a place to put it all, sometimes called the big bit bucket. This is like the investigative data mart for big databases, but more poly-structured. In some cases it is focused on data staging and transformation; but it can also be used for analysis in place.

The list of candidate technologies to run your bit bucket starts with Hadoop and Splunk.

Archival data store

• Kinds of data likely to be included: Operational, CDR (call detail record), security log
• Likely use styles: Archival, reporting (for compliance), possibly also investigative
• Examples: Any long-term detailed historical store
• Stresses: TCO, compression, scale-out, performance (if multi-use)

Analytic DBMS vendors have been insulting each other with the claim “that’s just an archival data store,” dating back at least to the first time Greenplum was deployed on an underpowered Sun Thumper system. Perhaps only Rainstor truly embraces the archival positioning, and I’ve become pretty dubious about their technical claims and their company alike.

Still, there’s a legitimate need for data stores — especially relational analytic DBMS that:

• Store data cheaply, with high rates of compression.
• Have decent performance if you do want to query the data.
• May have archiving/compliance-specific features as well.

Along with Rainstor, SAND and SenSage have at least partially targeted that use case. In addition, appliance vendors such as Teradata and Netezza try to have an archive-oriented product version in their lineups.

Outsourced data mart

• Kinds of data likely to be included: All
• Likely use styles: Traditional BI, investigative analytics, staging/ETL
• Examples: Advertising tracking, SaaS CRM
• Stresses: Performance, TCO, reliability, concurrency

Much of what happens in analytic database management can also be outsourced. Some applications that run via SaaS (Software as a Service) are analytic. I’ve had three different clients whose main business is picking marketing targets in various vertical segments; others who wanted to add analytics to what were historically OLTP applications; and others yet who just offered online business intelligence. Also, if your fundamental business is gathering data and reselling it to a variety of user organizations, that’s an analytic data management challenge. The possibilities expand from there.

Data outsourcers are in the IT business, and so their IT development is — hopefully! — more serious and less politically encumbered than at many conventional enterprises. Thus, legacy systems and master data management issues are commonly less prevalent, or at least more aggressively disposed of. The same, up to a point, goes for vendor politics.*  Multitenancy is commonly an issue, as is running in the cloud.

*Even so, there’s often That Guy who doesn’t want to migrate away from Oracle, no matter what.

Vertica gets the nod in a number of these cases; it’s cloud-friendly, and often the problem is naturally columnar. Other columnar products can be good choices too, with added brownie points for Infobright if the shop is MySQL-oriented anyway. Running Netezza or other appliances makes sense mainly if you’re pretty sure you want to keep operating your own data centers, but some data outsourcers are just fine with that assumption.

Operational analytic(s) server

• Kinds of data likely to be included: Customer-centric, log, financial trade
• Likely use styles: Advanced operational analytics
• Examples:
  • Lower latency: Web or call-center personalization, anti-fraud
  • Higher latency: Customer profiling, Basel 3 risk analysis
• Stresses: Performance, reliability, analytic functionality, perhaps concurrency

Even with eight different choices, I need a “catch-all” category; this is it.

Suppose you want to do reasonably sophisticated analytics, then use the results in operations. This is the classical challenge in integrating short-request and analytic processing. There are multiple ways to tackle it, embodying different trade-offs in cost, convenience, or analytic accuracy. If the platform on which you want to run your investigative analytics also has the reliability and concurrency appropriate for mission-critical operations, you’re set. Otherwise, you may want to pipe derived data into a more “industrial-strength” DBMS, ideally the one that runs your operational apps anyway

Another option is to integrate a limited amount of analytics immediately into your short-request processing system. For example, as bad as they are at the kinds of queries that require joins, NoSQL systems are often fast at simple aggregations. As MapReduce/NoSQL integrations mature, that option may not require pumping the data anywhere else for deeper analytics; even if it does, at least you’re starting out with the data in a convenient bit bucket.

Streaming/CEP-centric architectures could come into play as well. And it goes on from there. The possibilities in this last category are just too varied to generalize about.

So did I get them all? Or are there yet other analytic data management use cases that I don’t fit into my eight categories?

• Now we know (at least in part) why Vertica went oddly silent for a while.
• As per that same link, Vertica has >250 ordinary customers, and >70 more OEM sell-through ones.
• This is a setback for speculation about any kind of upcoming Aster/HP tie-up.
• Edit:  Forgot this one briefly — HP chairman Ray Lane was previously involved with Vertica.
• Vertica arguably is the most mature of the modern column-store DBMS — i.e., the ones that don’t have their roots in bitmaps the way Sybase and SAND do.
• HP executed really badly in data warehouse DBMS and appliances under former CEO Mark Hurd.
• Unfortunately, if you’re quickly researching Vertica, neither Gartner nor Forrester is a reliable source of detailed information.
• It would make sense for HP to acquire StreamBase too, and fold StreamBase into Vertica. Reasons include:
  • StreamBase and Vertica are aligned with each other. Both were founded by Mike Stonebraker, with overlapping groups of academic contributors. Both are in the Boston area. StreamBase and Vertica have worked together on various joint customer accounts, especially in the financial services sector.
  • Like other independent CEP vendors, StreamBase can’t or won’t accomplish much outside certain niches (mainly financial services).
  • StreamBase reports, rather credibly, that it’s doing well in its niches. While StreamBase’s success seems to include a heavy dose of professional services, that hardly would be a deal-breaker for HP.
• It would make partial sense for HP to acquire VoltDB, and fold VoltDB into Vertica.
  • VoltDB was actually spun out of Vertica, and incubated in Vertica offices. A lot of thinking has already been done about how to integrate VoltDB and Vertica.
  • VoltDB needs the help, as its strategy is not attuned to the needs of succeeding in a highly competitive, rapidly innovative marketplace.
  • VoltDB doesn’t have the kind of traction on which a big company like HP could hang an acquisition case or acquisition strategy.

That said,  

• Raj said that Sybase’s Aleri acquisition was, if anything, tracking ahead of expectations.
• Raj didn’t seem the slightest bit focused on the Coral8/Aleri CEP-based BI strategy that John Morell had long championed.
• Raj reminded me that Sybase SQL Anywhere has numerous OEMs, not just on the true desktop/laptop or smaller, but also in a return to its server/workgroup roots. Sybase SQL Anywhere even added geospatial indexing recently.

Raj also spoke glowingly of SAP’s in-memory database technology and the potential for Sybase of same — until I asked a follow-up question. At that point, he confessed that he didn’t really know much about about SAP’s in-memory database technology yet. As I said before, I believe SAP is fairly sincere about its belief that its in-memory database technology will conquer the world — but this is a naive and poorly-founded opinion even so.

One tidbit I did get is that SAP’s in-memory database technology is not just son-of-T-REX. A Korean (Raj thinks) company SAP had acquired is also in the mix. Raj also had the impression SAP’s in-memory technology can do rows, columns, or hybrid structures. On the one hand, that makes sense. On the other, it’s not a perfect fit with what Hasso Plattner said last year.

Meanwhile, Truviso — which never got much market traction in the first place — was in hiding; Roman Bukary never did keep his promise to brief me on the company’s new and improved strategy. Then Truviso had yet another management change, amidst rumors that it was repositioning away from CEP. As per a press release Truviso emailed today, that’s now official, with Truviso’s main business being something to do with web analytics.

• This is a good idea, and many software vendors should do it if they aren’t already.
• It’s no big deal.

For reasons why, let me quote an email I just sent to an inquiring reporter:

• StreamBase sells mainly to the financial services and intelligence community markets. Neither group will share much in the way of core algorithms.
• But both groups are pretty interested in open source software even so. (I think for both the price and customizability benefits.)
• Open source software commonly gets community contributions for connectors, adapters, and (national) language translations.
• But useful contributions in other areas are much rarer.
• Linden Labs is one of StreamBase’s few significant customers outside its two core markets.
• All of the above are consistent with the press release (which quotes only one StreamBase customer — guess who?).