• LiveView is a business intelligence and alerting suite built on/in the rest of StreamBase’s technology, meant to operate on streaming data.
• LiveView is positioned by StreamBase as having a true push event-driven architecture rather than pull/poll.
• StreamBase LiveView is designed to query in-memory data and then have the results change in real time as the data set changes.
• The LiveView user interface is a rapidly changing work in progress.
• LiveView has other Version 1 limitations as well
• LiveView is targeted squarely at StreamBase’s financial trading core market until some of the Version 1 limitations are lifted.

The basic StreamBase LiveView pipeline goes something like:  

• Data comes into the system via multiple streams.
• Transformations upon data arrival can include but are not limited to:
  • Aggregations.
  • Joins to reference data.
  • Joins to other streams.
• The streams (transformed or perhaps otherwise) are output to tables …
• … which are continuously updated as more data streams through.
• The data in the resulting table can be consumed:
  • Via LiveView-provided BI capabilities.
  • Via an API.

When wearing my vendor consultant hat, I warmly encourage StreamBase to emphasize the lack of a batch step anywhere in this process. As an analyst, however, I’m more restrained about a claim like “We uniquely free you from batch.” I agree that avoiding batch jobs is a Very Nice Thing. But you also are spared most batch-cycle processing if you stream updates from your short-request database to an analytic DBMS, e.g. via some kind of near-real-time replication.

That said, the push-versus-pull continuous filtering part of the StreamBase LiveView story seems pretty real. I think having sub-second display updates is cool in all sorts of BI use cases, and seriously useful in some number of them. While I don’t have a clear opinion as to whether the StreamBase approach offers huge performance advantages for that kind of latency over “pull” alternatives, my guess is in the direction of “yes”.

Version 1 limitations on StreamBase LiveView include:

• You consume data one table at a time, with no possibility of a join after the data has originally been put into a LiveView table.
• While LiveView in principle offers rich alerting potential, you get at it via an API rather than much in the way of alerting-specific tools.
• The first LiveView UI StreamBase put together looks a lot like 1980s stock quote machines. The next one it added looks a lot like Panopticon. Much cool-looking enhancement remains to be done.

One competitive (non)-note: This all sounds something like what TIBCO has been pushing for years, but in fact I don’t have much knowledge of TIBCO’s efforts in the area. I had a meeting set up to learn about it some time ago, but it got canceled because TIBCO’s PR people:

• Didn’t want to let any kind of meeting happen without them, even though a serious CTO-type representative seemed happy to talk, but also …
• … didn’t want to work at dinner time.

I haven’t had substantive contact with TIBCO since.

• StreamBase once planned to conquer the world.
• However, StreamBase really only sold effectively in the financial trading and intelligence markets.
• StreamBase retrenched, focusing almost exclusively on the financial trading market.
• With StreamBase LiveView, StreamBase is expanding from embedded operational analytics to do (also operational) business intelligence as well.
• StreamBase is hopeful that, perhaps starting with Version 2 or so, LiveView will be successful outside the financial trading market.

Not coincidental to these shifts in focus, StreamBase was our client, then stopped being one for a while, and now is a client again.

StreamBase (the product set) consists primarily of three things (LiveView aside):

• A development environment, whose output is in …
• … a visual programming language called EventFlow …
• … which is complied and executed by StreamBase’s execution layers.

One important set of ancillary products are StreamBase’s connectors to various data sources — StreamBase offers about 125 of its own, a number that approaches 200 when community contributions are included.

StreamBase has a second programming language called StreamSQL, but that’s rarely used except for embedding in or connecting to third-party software. EventFlow and StreamSQL compile to nearly identical byte code. (The main difference seems to be that as a practical matter you’ll name things a bit differently in the two languages, focusing on verbs in EventFlow and nouns in StreamSQL.)

StreamBase says that in the financial trading market, great performance out of the box equates to better time-to-value, since you are spared time you’d otherwise have to spend tuning the system. Implicit in that is a claim — which competitors might dispute — that StreamBase has great performance. StreamBase fondly thinks that having a domain-specific language gives it a leg up in achieving great compiler optimization. (The same would presumably apply to StreamBase’s competitors, but only if they have optimizing compilers themselves.)

One point that’s a little unusual for me these days is that StreamBase favors big SMP (Symmetric MultiProcessing) boxes over blade-based scale-out. 16+ cores and 256 gigabytes of RAM are not uncommon. Clusters commonly include 4-8 machines, but rarely more; the largest StreamBase cluster evidently contains 36 machines.

And with that I’ll turn to StreamBase’s newest offering, LiveView.

• The upshot of my what to call CEP thread in August was that “streaming” and “event processing” are not the same concept, but it so happens that they have the most traction where they intersect. That said, I both observe and endorse an apparent shift from “event” to “stream” as the core of the terminology, in a reversal of my opinion of several years ago.
• IBM continues to throw a lot of resources at its System S/ InfoSphere Streams product, but I haven’t heard yet of much marketplace success. That said, I believe IBM is still pretty serious about Streams, as one would expect from an effort whose code name so cheekily references System R. In particular, Streams shows up prominently on IBM’s top-level analytic architecture slide.
• Sybase recently released its ESP (Event Stream Processor) 5.0, which it says is the full merger of the Aleri and Coral8 predecessors. You can still get Sybase ESP without buying into the full Sybase RAP stack, and Sybase has no plans to change that.
• Sybase has discontinued all the business intelligence types of products Aleri and Coral8 were developing. Rather, Sybase is OEMing Panopticon, which it reports has been well received. Other than the discontinuation of the BI efforts, there seem to be few Aleri or Coral8 features missing from the merged Sybase ESP product.
• Truviso continues to be out of the picture.
• I have more to say about StreamBase separately.
• I have more to say about Sybase and IBM, which I’ll get to when I can.
• I have nothing new on Progress Apama. I also know little about any of the open source efforts.

Meanwhile, if you want to see technically nitty-gritty posts about the CEP/streaming area, you may want to look at my CEP/streaming coverage circa 2007-9, based on conversations with (among others) Mike Stonebraker, John Bates, and Mark Tsimelzon.

The other main alternative is to base the name on “stream processing” instead.* The CEP-or-whatever industry is split between these choices, with StreamBase currently favoring “CEP” (despite its company name), IBM emphatically favoring “stream”, and Sybase seemingly trying to have things both ways.

*And then, of course, there is “event stream processing”, regarding which please see below.

I’ve been juggling this terminological divide myself, referring to complex event/stream processing as long as four years ago. But enough is enough. I’d like to write more about the category without repeatedly apologizing for its name. And so, always bearing in mind Monash’s Third Law of Commercial Semantics, here’s where I’m coming down.

The more I think about it, the less I like the term “event processing”. Here’s why. Events happen; data is produced; CEP systems most commonly try to identify and categorize the events based on the data. The CEP systems may then do significant further processing, but more often they just pass the information on to another system (most commonly either persistent DBMS or “real-time” business intelligence). How much of that is really “event processing”? Relatively little, I’d say. And referring specifically to “complex” events doesn’t address my complaints at all.

So I’d like to go with some version of “stream”. But “stream processing” has other computer-related uses, while “Stream management” commonly describes care and planning for small waterways. So “stream” might do best with a modifier, such as “event” or “data”. Of the two, I prefer “data stream” (or “datastream”) to “event stream”; the events aren’t really streaming, but the data is.

So should it be “data stream processing” or “data stream management”? Well, the only one of numerous Wikipedia definitions I’ve actually liked while researching this post is the one for “Data Stream Management System“:

A Data Stream Management System (DSMS) is a set of computer programs that controls the maintenance and querying of data in data streams. The use of a DSMS to manage a data stream is roughly analogous to the use of a Database Management System (DBMS) to manage a conventional database.

A key feature of a DSMS is the ability to execute a continuous query against a data stream. A conventional database query executes once and returns a set of results for a given point in time. In contrast, a continuous query continues to execute over time, as new data enters the stream. The results of the continuous query are updated as new data appears.

I think the data stream/database management analogy is spot on. Your queries work a little differently, but otherwise you’re doing pretty much the same things. Indeed, you’re probably even going to persistently store some of the data, and ideally that DBMS capability would be tightly integrated into your CEP system. (In practice they’re apt to be more loosely coupled; for most purposes that works well enough.) Query execution, data ingestion, performance monitoring/tuning, workload prioritization — it’s very DBMS-like stuff. And by the way, “data stream management system” is the term that was used by the researchers — Mike Stonebreaker, Stan Zdonik, Dan Abadi, et al. — who wrote a paper describing the project on which StreamBase was based … although some might question whether that particular observation is a strong signal of accuracy.

This reasoning suggests Data Stream Management System is what it should be. The usual kinds of abbreviation — datastream (product), datastream manager, DSMS, etc. would no doubt follow. So should it be “Data Stream”, “Datastream”, or “Data-stream”? At that level of detail, I don’t yet have an opinion.

The only thing is — that’s all pretty wordy compared to CEP. So after all this, I’m still not sure which term(s) I prefer.

What are your thoughts?

• Now we know (at least in part) why Vertica went oddly silent for a while.
• As per that same link, Vertica has >250 ordinary customers, and >70 more OEM sell-through ones.
• This is a setback for speculation about any kind of upcoming Aster/HP tie-up.
• Edit:  Forgot this one briefly — HP chairman Ray Lane was previously involved with Vertica.
• Vertica arguably is the most mature of the modern column-store DBMS — i.e., the ones that don’t have their roots in bitmaps the way Sybase and SAND do.
• HP executed really badly in data warehouse DBMS and appliances under former CEO Mark Hurd.
• Unfortunately, if you’re quickly researching Vertica, neither Gartner nor Forrester is a reliable source of detailed information.
• It would make sense for HP to acquire StreamBase too, and fold StreamBase into Vertica. Reasons include:
  • StreamBase and Vertica are aligned with each other. Both were founded by Mike Stonebraker, with overlapping groups of academic contributors. Both are in the Boston area. StreamBase and Vertica have worked together on various joint customer accounts, especially in the financial services sector.
  • Like other independent CEP vendors, StreamBase can’t or won’t accomplish much outside certain niches (mainly financial services).
  • StreamBase reports, rather credibly, that it’s doing well in its niches. While StreamBase’s success seems to include a heavy dose of professional services, that hardly would be a deal-breaker for HP.
• It would make partial sense for HP to acquire VoltDB, and fold VoltDB into Vertica.
  • VoltDB was actually spun out of Vertica, and incubated in Vertica offices. A lot of thinking has already been done about how to integrate VoltDB and Vertica.
  • VoltDB needs the help, as its strategy is not attuned to the needs of succeeding in a highly competitive, rapidly innovative marketplace.
  • VoltDB doesn’t have the kind of traction on which a big company like HP could hang an acquisition case or acquisition strategy.

• This is a good idea, and many software vendors should do it if they aren’t already.
• It’s no big deal.

For reasons why, let me quote an email I just sent to an inquiring reporter:

• StreamBase sells mainly to the financial services and intelligence community markets. Neither group will share much in the way of core algorithms.
• But both groups are pretty interested in open source software even so. (I think for both the price and customizability benefits.)
• Open source software commonly gets community contributions for connectors, adapters, and (national) language translations.
• But useful contributions in other areas are much rarer.
• Linden Labs is one of StreamBase’s few significant customers outside its two core markets.
• All of the above are consistent with the press release (which quotes only one StreamBase customer — guess who?).

• Most independent CEP vendors have some kind of application story in the capital markets vertical, such as packaged applications, ISV partners with packaged applications, application frameworks, and so on.
• CEP vendors offer lots of connectors to specific financial industry price/quote/trade feeds, as well as the usual other kinds of database connectivity (SQL, XML, etc.)
• Aleri/Coral8 (separately and now together) like to call attention to their business intelligence/analytics offerings. Analytics is front-and-center on Truviso’s web site too, not that Truviso does much to call attention to itself, period. (Roman Bukary once said he’d outline Truviso’s new strategy to me in 6-8 weeks or so … it’s now 14 months and counting.)

So far as I can tell, the areas of applications and analytics are fairly uncontroversial. Different CEP vendors have implemented different kinds of things, no doubt focusing on those they thought they would find easiest to build and then sell. But these seem to be choices in business execution, not in core technical philosophy.

In CEP application development, however, real philosophical differences do seem to arise. There are at least three different CEP application development paradigms:

• DML (Data Manipulation Language) extensions to handle time windows, etc., which are then embodied into a conventional application development stack. Microsoft exemplifies an extreme form of this strategy, but most vendors offer it at least as an option.
• Visual event-oriented programming language. This is StreamBase’s claim to fame; StreamBase says that 95% of its customers do 100% of their development in its Eclipse-based visual tool, which truly creates executable programs without any kind of code generation step. (Other “this isn’t just a toy” buzzwords StreamBase offered were “modularity, ” “parametrizability,” and, best of all, “visual debugger.”)
• Rules engine. When Progress Apama first told me how its preferred tool (I think a precursor of what is now Apama Event Modeler) worked, it sounded a lot like a RETE-based expert system shell. The Apama folks assured me that this really wasn’t RETE, and I see that there’s definitely more than just a rules language in Apama MonitorScript. Still, the paradigm seems to basically be that you write a bunch of rules, which are then executed by a suitable engine.

Even more fundamental, however, is the question of event-driven programming. That’s a term which first was popular in the 1990s, signifying programs that — rather than being wholly procedural — listened for and responded to events, often in user interfaces. More generally, event-driven programming is inherent in almost any kind of loosely-coupled architecture, be it client-server, service-oriented, or whatever. But to hear some CEP proponents tell it, all that isn’t event-driven enough.

In particular, StreamBase recommended to me Gregor Hohpe’s paper Programming Without a Call Stack – Event-driven Architectures, which argues that if there’s any concept of procedure or method invocation at all, the whole thing is too “command-and-control” (Boo!!) and insufficiently event-driven to be well-suited for CEP. Rather, everything should be done on a fine-grained publish-subscribe basis. It’s an interesting argument. Usually, the problem with extremist programming paradigms is that the thing you write has to interface with the rest of the world, and by the time it accommodates itself to them, the paradigm is violated anyway. But if the essence of the paradigm is loose coupling to begin with, maybe that pitfall can for once be avoided.

]]> http://www.dbms2.com/2009/05/21/notes-on-cep-application-development/feed/ 5 http://www.dbms2.com/2009/05/21/notes-on-cep-performance/ http://www.dbms2.com/2009/05/21/notes-on-cep-performance/#comments Thu, 21 May 2009 08:20:36 +0000 Curt Monash http://www.dbms2.com/?p=787 I’ve been talking to CEP vendors on and off for a few years. So what I hear about performance is fairly patchwork. On the other hand, maybe 1-2+ year-old figures of per-core performance are still meaningful today. After all, Moore’s Law is being reflected more in core count than per-core performance, and it seems CEP vendors’ development efforts haven’t necessarily been concentrated on raw engine speed.

So anyway, what do you guys have to add to the following observations?

• Super-low-latency financial services industry tasks are often “embarrassingly parallel.” Thus, near-linear scale-out is common.
• That said, good parallelism seems fairly new in CEP engines (of course, CEP engines are fairly new themselves — for all I know, some have been parallel since inception).
• I’ve heard claims of up to 400,000 messages/second/core for simple queries or patterns.
• I’ve heard claims of 70,000 messages/core for not-so-simple queries or patterns, and probably higher than that depending on what the meaning of “simple” is.
• IBM just disclosed >15,000 messages/core on a pretty low-powered processor.
• I’ve heard that Coral8, Apama, and StreamBase rarely lost deals due to performance or throughput problems. I’ve heard that the same is not as true of Aleri.
• StreamBase proudly says it’s been fully multithreaded since academic research-project days.  For Apama multithreading is evidently a more recent feature. But does it matter much?

• StreamBase once planned to conquer the world, making an impact as big as database management’s. Now it has retreated into niche markets.
• Progress Software, a decent-sized company, put a large fraction of its energy into Apama. Little has happened outside the financial service sector.
• Coral8 has some great-sounding ideas. But Coral8 now has merged into Aleri, basically a financial-markets specialist.
• Mike Franklin says some ambitious things on behalf of Truviso, but I haven’t noticed much traction there either.

CEP’s penetration outside of its classical markets isn’t quite zero. Customers include several transportation companies (various vendors), Sallie Mae (Coral8), a game vendor or two (StreamBase, if I recall correctly), Verizon (Aleri, I think), and more. But I just wrote that list from memory — based mainly on not-so-recent deals — and a quick tour of the vendors’ web sites hasn’t turned up much I overlooked. (Truviso does have a recent deal with Technorati, but that’s not exactly a blue chip customer these days.)

So far as I can tell, this is a new version of a repeated story. A clever alternative to relational DBMS was invented. It proved superior in some specific applications and vertical markets. It failed to achieve much broader adoption. Initial high hopes got dashed, companies failed to grow rapidly, and shareholders grew tired.

So where will things go from here? My best guesses include:

• The financial trading market isn’t going away for CEP. Super-low-latency is really needed there.
• As much as I love the idea of CEP-infused BI, it will be adopted only at the rate broader-based BI vendors can support.
• A few niches will generate some business for CEP in data reduction. Leading candidates are the ones where there’s been a little traction to date — national intelligence, transportation, web analytics, and so on.
• Sadly, that’s about it.

]]> http://www.dbms2.com/2009/03/09/independent-cep-vendors-continue-to-flounder/feed/ 9 http://www.dbms2.com/2008/01/16/twitter-could-easily-be-made-reliable/ http://www.dbms2.com/2008/01/16/twitter-could-easily-be-made-reliable/#comments Wed, 16 Jan 2008 22:52:22 +0000 Curt Monash http://www.dbms2.com/2008/01/16/twitter-could-easily-be-made-reliable/ There’s a lot of agitation today because Twitter broke under the message volume generated during Steve Jobs’ Macworld keynote. I don’t know what that volume was, but I just checked the lower volume of tweets (i.e., updates) going through the “public timeline” (i.e., everything) twice, and both times it was under 200 messages per minute. So, let’s say there’s a much higher volume at peak times, and also hypothesize that Twitter would like to grow a lot, and say that Twitter would like to handle 10-100,000 messages/minute – i.e., 1000+/second — as soon as possible.

That’s easy using CEP (Complex Event Processing). A Twitter update is just a string of 140 or fewer characters. It is associated with three pieces of metadata – author, time, and mode of posting. It should be visible in real time to any of the author’s “followers,” as well as in a single public timeline; perhaps there will be other kinds of Twitter channels in the future. In most cases, these updates are only visible to a user upon page refresh. Almost nNo Twitter user seems to have more than about 7,000 followers, even Robert Scoble or Evan Williams.* The average number of followers, at least among active updaters, is probably in the low hundreds now. So basically, this is all a heckuva lot easier than the tick-monitoring systems Wall Street firms are using today.

*I believe there’s a hard cap of 7,500, but nobody seems to have bumped against it yet.Twitterholic gives a different figure than Twitter does for Scoble. And it correctly shows Dave Troy with a little over 10,000.

Here’s how to implement that. You start with a complex event/stream processing engine like StreamBase or Coral8, both of which can handle a large multiple of that volume without working up a sweat. And you basically do everything in RAM. 6 million messages per hour of under 200 bytes each? Not a lot of RAM needed for that. A set of 25 or so recent messages cached for each of, say, the 100,000 or 1 million most recent users? Also not a whole lot of RAM. And if you push out aged messages and replace them with more recent message IDs, the cache gets smaller. Banging things to disk for persistence is an exercise left to the reader.

Meanwhile, Dave Winer views Twitter as a distributed computing challenge and Larry Dignan asks whether Twitter needs to be reliable at all (he leans to “Yes” because of the new apps that would open up).