Complex event processing (CEP)
Discussion of complex event processing (CEP), aka event processing or stream processing – i.e., of technology that executes queries before data is ever stored on disk. Related subjects include:
After posting about IBM’s System S/InfoSphere Streams CEP offering, I sent three followup questions over to Jeff Jones. It seems simplest to just post the Q&A verbatim.
1. Just how many processors or cores does it take to get those 5 million messages/sec through? A little birdie says 4,000 cores. Read more
|Categories: Analytic technologies, Complex event processing (CEP), IBM and DB2, Investment research and trading||7 Comments|
Microsoft still hasn’t worked out all the kinks regarding when and how intensely to brief me. So most of what I know about their announcement earlier this week of a CEP/stream processing product* is what I garnered on a consulting call in March. That said, I sent Microsoft my notes from that call, they responded quickly and clearly to my question as to what remained under NDA, and for good measure they included a couple of clarifying comments that I’ll copy below.
*”in the SQL Server 2008 R2 timeframe,” about which Microsoft wrote “the first Community Technology Preview (CTP) of SQL Server 2008 R2 will be available for download in the second half of 2009 and the release is on track to ship in the first half of calendar year 2010. “
Perhaps it is more than coincidence that IBM rushed out its own announcement of an immature CEP technology — due to be more mature in a 2010 release — immediately after Microsoft revealed its plans. Anyhow, taken together, these announcements support my theory that the small independent CEP/stream processing vendors are more or less ceding broad parts of the potential stream processing market.
The main use cases Microsoft talks about for CEP are in the area of sensor data. Read more
|Categories: Analytic technologies, Application areas, Complex event processing (CEP), Microsoft and SQL*Server||8 Comments|
IBM has hastily announced System S Streams, a product that was supposed to be called InfoSphere Streams and introduced only in 2010. Apparently, the rush is because senior management wanted to talk about it later this week, and perhaps also because it was implicitly baked into some of IBM’s advertising already. Scrambling ensued. Even so, Jeff Jones and team got to me fast, and briefed me — fairly non-technically, unfortunately, but otherwise how I like it, namely on a harmless embargo and without any NDAs. That’s more than can be said for my clients at Microsoft, who also introduced CEP this week, but I digress …
*Indeed, as I draft this post-Celtics-game, the embargo is already expired.
Marketing aside, IBM System S/InfoSphere Streams is indeed a CEP/stream processing engine + language (with an Eclipse-based development environment). Apparently, IBM’s thinks InfoSphere Streams (if that’s what it winds up being renamed to) is or will be differentiated from other CEP packages in:
- Scale-out. (That’s the one that appears to be real today. In fact, there’s a prototype running on Blue Gene.)
- Support for complex datatypes such as XML, text, voice, video, etc.
- Security and general industrial-strengthness.
|Categories: Analytic technologies, Application areas, Complex event processing (CEP), IBM and DB2, Investment research and trading, Scientific research||3 Comments|
My skeptical remarks on the Aleri/Coral8 merger generated some pushback. Today I actually got around to talking with John Morell, who was marketing chief at Coral8 and has remained with the combined company. First, some quick metrics:
- The combined Aleri has around 100 employees, 60-40 from Aleri vs. Coral8.
- The combined Aleri has around 80 customers. All of Aleri’s, with one sort-of exception at Banks.com, were in financial services. A large minority of Coral8’s were in financial services too.
- However, half of Aleri’s marketing spend going forward is budgeted outside the financial services markets. Not unreasonably, John presents this as a proof point Aleri is serious about selling to other markets.
- Aleri had 12-14 people in the UK pre-merger. Coral8 had none in Europe.
- Coral8 had 15 OEMs pre-merger, some actually generating revenue. Aleri had substantially none.
- Coral8 had been closing a “couple” of customers/quarter in online commerce. But recently, that rate ramped up to a “few.”
- Aleri’s engine is used to handle “many” hundreds of thousands of messages per second. Coral8’s highest-throughput user processes 100-150,000 messages/second.
John is sticking by the company line that there will be an integrated Aleri/Coral8 engine in around 12 months, with all the performance optimization of Aleri and flexibility of Coral8, that compiles and runs code from any of the development tools either Aleri or Coral8 now has. While this is a lot faster than, say, the Informix/Illustra or Oracle/IRI Express integrations, John insists that integrating CEP engines is a lot easier. We’ll see.
I focused most of the conversation on Aleri’s forthcoming efforts outside the financial services market. John sees these as being focused around Coral8’s old “Continuous (Business) Intelligence” message, enhanced by Aleri’s Live OLAP. Aleri Live OLAP is an in-memory OLAP engine, real-time/event-driven, fed by CEP. Queries can be submitted via ODBO/MDX today. XMLA is coming. John reports that quite a few Coral8 customers are interested in Live OLAP, and positions the capability as one Coral8 would have had to develop had the company remained independent. Read more
|Categories: Aleri and Coral8, Analytic technologies, Application areas, Complex event processing (CEP), Games and virtual worlds, Investment research and trading, MOLAP, Web analytics||4 Comments|
I’m not aware of anyone claiming “CEP is an alternative to a relational RDBMS” – except maybe as an application platform for processing events (where RDBMS could be seen as a square hole for an event round peg).
Actually, it’s hard to think of an application for off-the-shelf CEP where the alternative technologies aren’t:
- Custom CEP
- Just write it to an RDBMS and query it
Independent CEP (Complex/Event Processing) vendors continue to flounder, at least outside the financial services and national intelligence markets.
- StreamBase once planned to conquer the world, making an impact as big as database management’s. Now it has retreated into niche markets.
- Progress Software, a decent-sized company, put a large fraction of its energy into Apama. Little has happened outside the financial service sector.
- Coral8 has some great-sounding ideas. But Coral8 now has merged into Aleri, basically a financial-markets specialist.
- Mike Franklin says some ambitious things on behalf of Truviso, but I haven’t noticed much traction there either.
CEP’s penetration outside of its classical markets isn’t quite zero. Customers include several transportation companies (various vendors), Sallie Mae (Coral8), a game vendor or two (StreamBase, if I recall correctly), Verizon (Aleri, I think), and more. But I just wrote that list from memory — based mainly on not-so-recent deals — and a quick tour of the vendors’ web sites hasn’t turned up much I overlooked. (Truviso does have a recent deal with Technorati, but that’s not exactly a blue chip customer these days.)
So far as I can tell, this is a new version of a repeated story. Read more
|Categories: Aleri and Coral8, Analytic technologies, Business intelligence, Complex event processing (CEP), Progress, Apama, and DataDirect, StreamBase, Truviso||12 Comments|
When I went to Oracle in October, the main purpose of the visit was to discuss Exadata. And so my initial post based on the visit was focused accordingly. But there were a number of other interesting points I’ve never gotten around to writing up. Let me now remedy that, at least in part. Read more
|Categories: Complex event processing (CEP), Data types, Data warehousing, Database compression, GIS and geospatial, MOLAP, Oracle, SAP AG, Theory and architecture, Web analytics||9 Comments|
It used to be that Coral8 and StreamBase were the two complex event/stream processing (CEP) vendors most committed to branching out beyond the super-low-latency algorithmic trading marketing. But StreamBase seems to have pulled in its horns after a management change, focusing much more on the financial market (and perhaps the defense/intelligence market as well). Aleri, Truviso, and Progress Apama, while each showing signs of branching out, don’t seem to have gone as far as Coral8 yet. And so, though it’s a small company with not all that many dozens of customers, my client Coral8 seems to be the one to look at when seeing whether CEP really is relevant to a broad range of mainstream – no pun intended – applications.
Coral8 today unveiled a new product release – the not-so-concisely named “Coral8 Engine and Portal Release 5.5” – and a new buzzphrase — “Continuous Intelligence.” The interesting part boils down to this:
Coral8 is proposing CEP — excuse me, “Continuous Intelligence” — as a data-store-equivalent for business intelligence.
This includes both operational BI (the current sweet spot) and dashboards (the part with cool, real-time-visualization demos). Read more
|Categories: Aleri and Coral8, Analytic technologies, Application areas, Business intelligence, Complex event processing (CEP), Data warehousing, Investment research and trading, Memory-centric data management, Web analytics||12 Comments|
It should surprise nobody that web analytics – and specifically clickstream data — is one of the biggest areas for high-end data warehousing. For example:
- I believe that both of the previously mentioned petabyte+ databases on Greenplum will feature clickstream data.
- Aster Data’s largest disclosed database, by almost two orders of magnitude, is at MySpace.
- Clickstream analytics is a big application area for Vertica Systems.
- Clickstream analytics is a big application area for Netezza.
- Infobright’s customer success stories appear to be concentrated in clickstream analytics.
- Coral8 tells me that CEP is also being used for clickstream data, although I suspect that a lot of Coral8’s evidence in that regard comes from a single flagship account. Edit: Actually, Coral8 has a bunch of clickstream customers.
|Categories: Aleri and Coral8, Aster Data, Complex event processing (CEP), Greenplum, Infobright, Netezza, Vertica Systems, Web analytics||2 Comments|
Doug Henschen has two good articles based on Gartner’s Event Processing conference, on the theme of BI/event processing integration — an overview, and a detailed interview with Roy Schulte. And as I note elsewhere, Seth Grimes has a good article based on the conference too.
I have my own thoughts on these subjects, but I’m not ready to post them at the moment. In the mean time, I recommend the articles linked above.
|Categories: Analytic technologies, Business intelligence, Complex event processing (CEP)||Leave a Comment|