StreamBase
Analysis of complex event/stream processing vendor StreamBase. Related subjects include:
StreamBase LiveView — push-based real-time BI
My clients at StreamBase are coming out with a new product line called LiveView, and I agreed they could launch it via this blog. Key points about StreamBase LiveView Version 1.0 include:
- LiveView is a business intelligence and alerting suite built on/in the rest of StreamBase’s technology, meant to operate on streaming data.
- LiveView is positioned by StreamBase as having a true push event-driven architecture rather than pull/poll.
- StreamBase LiveView is designed to query in-memory data and then have the results change in real time as the data set changes.
- The LiveView user interface is a rapidly changing work in progress.
- LiveView has other Version 1 limitations as well
- LiveView is targeted squarely at StreamBase’s financial trading core market until some of the Version 1 limitations are lifted.
The basic StreamBase LiveView pipeline goes something like: Read more
| Categories: Business intelligence, Complex event processing (CEP), Data warehousing, Memory-centric data management, StreamBase | 1 Comment |
StreamBase catchup
While I was cryptic in my general CEP/streaming catchup, I’ll say a bit more regarding StreamBase in particular. At the highest level, non-technically:
- StreamBase once planned to conquer the world.
- However, StreamBase really only sold effectively in the financial trading and intelligence markets.
- StreamBase retrenched, focusing almost exclusively on the financial trading market.
- With StreamBase LiveView, StreamBase is expanding from embedded operational analytics to do (also operational) business intelligence as well.
- StreamBase is hopeful that, perhaps starting with Version 2 or so, LiveView will be successful outside the financial trading market.
| Categories: Complex event processing (CEP), Investment research and trading, Parallelization, StreamBase | 2 Comments |
Very brief CEP/streaming catchup
When I agreed to launch the StreamBase LiveView product via DBMS 2, I planned to catch up on the whole CEP/streaming area first. Due to the power and internet outages last week, that didn’t entirely happen. So I’ll do a bit of that now, albeit more cryptically than I hoped and intended.
- The upshot of my what to call CEP thread in August was that “streaming” and “event processing” are not the same concept, but it so happens that they have the most traction where they intersect. That said, I both observe and endorse an apparent shift from “event” to “stream” as the core of the terminology, in a reversal of my opinion of several years ago.
- IBM continues to throw a lot of resources at its System S/ InfoSphere Streams product, but I haven’t heard yet of much marketplace success. That said, I believe IBM is still pretty serious about Streams, as one would expect from an effort whose code name so cheekily references System R. In particular, Streams shows up prominently on IBM’s top-level analytic architecture slide.
- Sybase recently released its ESP (Event Stream Processor) 5.0, which it says is the full merger of the Aleri and Coral8 predecessors. You can still get Sybase ESP without buying into the full Sybase RAP stack, and Sybase has no plans to change that.
- Sybase has discontinued all the business intelligence types of products Aleri and Coral8 were developing. Rather, Sybase is OEMing Panopticon, which it reports has been well received. Other than the discontinuation of the BI efforts, there seem to be few Aleri or Coral8 features missing from the merged Sybase ESP product.
- Truviso continues to be out of the picture.
- I have more to say about StreamBase separately.
- I have more to say about Sybase and IBM, which I’ll get to when I can.
- I have nothing new on Progress Apama. I also know little about any of the open source efforts.
Meanwhile, if you want to see technically nitty-gritty posts about the CEP/streaming area, you may want to look at my CEP/streaming coverage circa 2007-9, based on conversations with (among others) Mike Stonebraker, John Bates, and Mark Tsimelzon.
| Categories: Business intelligence, Complex event processing (CEP), IBM and DB2, StreamBase, Sybase, Truviso | 1 Comment |
Renaming CEP … or not
One of the less popular category names I deal with is “Complex Event Processing (CEP)”. The word “complex” looks weird, and many are unsure about the “event processing” part as well. CEP does have one virtue as a name, however — it’s concise.
The other main alternative is to base the name on “stream processing” instead.* The CEP-or-whatever industry is split between these choices, with StreamBase currently favoring “CEP” (despite its company name), IBM emphatically favoring “stream”, and Sybase seemingly trying to have things both ways.
*And then, of course, there is “event stream processing”, regarding which please see below.
| Categories: Complex event processing (CEP), StreamBase | 24 Comments |
Some quick notes on HP-Vertica
HP is acquiring Vertica. Read more
| Categories: Complex event processing (CEP), In-memory DBMS, Investment research and trading, Memory-centric data management, StreamBase, VoltDB and H-Store | 12 Comments |
Quick thoughts on the StreamBase Component Exchange
Streambase is announcing something called the StreamBase Component Exchange, for developers to exchange components to be used with the StreamBase engine, presumably on an open source basis. I simultaneously think:
- This is a good idea, and many software vendors should do it if they aren’t already.
- It’s no big deal.
For reasons why, let me quote an email I just sent to an inquiring reporter:
- StreamBase sells mainly to the financial services and intelligence community markets. Neither group will share much in the way of core algorithms.
- But both groups are pretty interested in open source software even so. (I think for both the price and customizability benefits.)
- Open source software commonly gets community contributions for connectors, adapters, and (national) language translations.
- But useful contributions in other areas are much rarer.
- Linden Labs is one of StreamBase’s few significant customers outside its two core markets.
- All of the above are consistent with the press release (which quotes only one StreamBase customer — guess who?).
| Categories: Complex event processing (CEP), Games and virtual worlds, Investment research and trading, Open source, StreamBase | 5 Comments |
Notes on CEP application development
While performance may not be all that great a source of CEP competitive differentiation, event processing vendors find plenty of other bases for technological competition, including application development, analytics, packaged applications, and data integration. In particular:
- Most independent CEP vendors have some kind of application story in the capital markets vertical, such as packaged applications, ISV partners with packaged applications, application frameworks, and so on.
- CEP vendors offer lots of connectors to specific financial industry price/quote/trade feeds, as well as the usual other kinds of database connectivity (SQL, XML, etc.)
- Aleri/Coral8 (separately and now together) like to call attention to their business intelligence/analytics offerings. Analytics is front-and-center on Truviso’s web site too, not that Truviso does much to call attention to itself, period. (Roman Bukary once said he’d outline Truviso’s new strategy to me in 6-8 weeks or so … it’s now 14 months and counting.)
So far as I can tell, the areas of applications and analytics are fairly uncontroversial. Different CEP vendors have implemented different kinds of things, no doubt focusing on those they thought they would find easiest to build and then sell. But these seem to be choices in business execution, not in core technical philosophy.
In CEP application development, however, real philosophical differences do seem to arise. There are at least three different CEP application development paradigms: Read more
| Categories: Aleri and Coral8, Business intelligence, Complex event processing (CEP), Microsoft and SQL*Server, Progress, Apama, and DataDirect, StreamBase | 5 Comments |
Notes on CEP performance
I’ve been talking to CEP vendors on and off for a few years. So what I hear about performance is fairly patchwork. On the other hand, maybe 1-2+ year-old figures of per-core performance are still meaningful today. After all, Moore’s Law is being reflected more in core count than per-core performance, and it seems CEP vendors’ development efforts haven’t necessarily been concentrated on raw engine speed.
So anyway, what do you guys have to add to the following observations?
- Super-low-latency financial services industry tasks are often “embarrassingly parallel.” Thus, near-linear scale-out is common.
- That said, good parallelism seems fairly new in CEP engines (of course, CEP engines are fairly new themselves — for all I know, some have been parallel since inception).
- I’ve heard claims of up to 400,000 messages/second/core for simple queries or patterns.
- I’ve heard claims of 70,000 messages/core for not-so-simple queries or patterns, and probably higher than that depending on what the meaning of “simple” is.
- IBM just disclosed >15,000 messages/core on a pretty low-powered processor.
- I’ve heard that Coral8, Apama, and StreamBase rarely lost deals due to performance or throughput problems. I’ve heard that the same is not as true of Aleri.
- StreamBase proudly says it’s been fully multithreaded since academic research-project days. For Apama multithreading is evidently a more recent feature. But does it matter much?
| Categories: Aleri and Coral8, Complex event processing (CEP), IBM and DB2, Memory-centric data management, Progress, Apama, and DataDirect, StreamBase | 13 Comments |
Independent CEP vendors continue to flounder
Independent CEP (Complex/Event Processing) vendors continue to flounder, at least outside the financial services and national intelligence markets.
- StreamBase once planned to conquer the world, making an impact as big as database management’s. Now it has retreated into niche markets.
- Progress Software, a decent-sized company, put a large fraction of its energy into Apama. Little has happened outside the financial service sector.
- Coral8 has some great-sounding ideas. But Coral8 now has merged into Aleri, basically a financial-markets specialist.
- Mike Franklin says some ambitious things on behalf of Truviso, but I haven’t noticed much traction there either.
CEP’s penetration outside of its classical markets isn’t quite zero. Customers include several transportation companies (various vendors), Sallie Mae (Coral8), a game vendor or two (StreamBase, if I recall correctly), Verizon (Aleri, I think), and more. But I just wrote that list from memory — based mainly on not-so-recent deals — and a quick tour of the vendors’ web sites hasn’t turned up much I overlooked. (Truviso does have a recent deal with Technorati, but that’s not exactly a blue chip customer these days.)
So far as I can tell, this is a new version of a repeated story. Read more
| Categories: Aleri and Coral8, Analytic technologies, Business intelligence, Complex event processing (CEP), Progress, Apama, and DataDirect, StreamBase, Truviso | 9 Comments |
Fixing Twitter in three letters: CEP
There’s a lot of agitation today because Twitter broke under the message volume generated during Steve Jobs’ Macworld keynote. I don’t know what that volume was, but I just checked the lower volume of tweets (i.e., updates) going through the “public timeline” (i.e., everything) twice, and both times it was under 200 messages per minute. So, let’s say there’s a much higher volume at peak times, and also hypothesize that Twitter would like to grow a lot, and say that Twitter would like to handle 10-100,000 messages/minute – i.e., 1000+/second — as soon as possible.
That’s easy using CEP (Complex Event Processing). A Twitter update is just a string of 140 or fewer characters. It is associated with three pieces of metadata – author, time, and mode of posting. It should be visible in real time to any of the author’s “followers,” as well as in a single public timeline; perhaps there will be other kinds of Twitter channels in the future. In most cases, these updates are only visible to a user upon page refresh. Almost nNo Twitter user seems to have more than about 7,000 followers, even Robert Scoble or Evan Williams.* The average number of followers, at least among active updaters, is probably in the low hundreds now. So basically, this is all a heckuva lot easier than the tick-monitoring systems Wall Street firms are using today.
*I believe there’s a hard cap of 7,500, but nobody seems to have bumped against it yet.Twitterholic gives a different figure than Twitter does for Scoble. And it correctly shows Dave Troy with a little over 10,000.
Here’s how to implement that. Read more