Aleri and Coral8
Analysis of complex event/stream processing vendors Aleri and Coral8 (now merged). Related subjects include:
CEP is entering BI
I talked with both Coral8 and Truviso this afternoon. They both have their financial services efforts, of course. Coral8 also continues to get business doing data reduction for sensor networks — mainly RFID and utilities, I think. Coral8 is working on some really cool and confidential other stuff as well.
But my biggest takeaway from this pair of calls was that Coral8 and Truviso are penetrating general BI. Read more
| Categories: Aleri and Coral8, Analytic technologies, Business intelligence, Memory-centric data management, Streaming and complex event processing (CEP), Truviso | Leave a Comment |
Fixing Twitter in three letters: CEP
There’s a lot of agitation today because Twitter broke under the message volume generated during Steve Jobs’ Macworld keynote. I don’t know what that volume was, but I just checked the lower volume of tweets (i.e., updates) going through the “public timeline” (i.e., everything) twice, and both times it was under 200 messages per minute. So, let’s say there’s a much higher volume at peak times, and also hypothesize that Twitter would like to grow a lot, and say that Twitter would like to handle 10-100,000 messages/minute – i.e., 1000+/second — as soon as possible.
That’s easy using CEP (Complex Event Processing). A Twitter update is just a string of 140 or fewer characters. It is associated with three pieces of metadata – author, time, and mode of posting. It should be visible in real time to any of the author’s “followers,” as well as in a single public timeline; perhaps there will be other kinds of Twitter channels in the future. In most cases, these updates are only visible to a user upon page refresh. Almost nNo Twitter user seems to have more than about 7,000 followers, even Robert Scoble or Evan Williams.* The average number of followers, at least among active updaters, is probably in the low hundreds now. So basically, this is all a heckuva lot easier than the tick-monitoring systems Wall Street firms are using today.
*I believe there’s a hard cap of 7,500, but nobody seems to have bumped against it yet.Twitterholic gives a different figure than Twitter does for Scoble. And it correctly shows Dave Troy with a little over 10,000.
Here’s how to implement that. Read more
| Categories: Aleri and Coral8, Memory-centric data management, StreamBase, Streaming and complex event processing (CEP) | 13 Comments |
Intelligent Enterprise’s list of 12/36/48 vendors
I’m getting a flood of press releases today, because many of the companies I write about were selected to Intelligent Enterprise’s list of 12 most influential vendors plus 36 more to watch in the areas Intelligent Enterprise covers (which seems to be pretty much the analytics-related parts of what I write about here and on Text Technologies). It looks like a pretty reasonable list, although I think they forced the issue in some of the small analytics vendors they selected, and of course anybody can quibble with some of the omissions.
Among the companies they cited, you can find topical categories here for IBM (and Cognos), Informatica, Microsoft, Netezza, Oracle, SAP/Business Objects (both), SAS, and Teradata; QlikTech; Cast Iron, Coral8, DATAllegro, HP, ParAccel, and StreamBase; and Software AG. On Text Technologies you’ll find categories for some of the same vendors, plus Attensity, Clarabridge, and Google. There also are categories for some of these vendors on the Monash Report.
Coral8 highlights some key issues with dashboards
Coral8 today is rolling out the Coral8 Portal, offering some BI basics for CEP (Complex Event Processing) filters and queries. In Release 1, this is primitive compared with other BI portals, and of direct interest only to organizations that have already decided they’re using CEP technology. Even so, it serves as a useful illustration of several important issues in dashboarding.
The simplest is that real-time dashboards require different visualizations than others. Most obvious is the ever-popular graph marching from right to left across the screen as time advances along the x-axis. There also are difference in styles between reports and tables that you actually read, vs. read-outs that you merely watch for flickers of change. (Of course those two examples hardly make for a complete list.)
More interesting is the flexibility and parameterization. While Coral8 sells to multiple markets, the design point for the portal is clearly financial trading. So, for example, a query may be registered with one ticker symbol, and an end user can easily customize it to slot in another one instead. In a way, this is a step toward the much greater flexibility that dashboards need overall.
Truth be told, if you put all such Coral8 flexibility features together they’re not yet very impressive. So what’s even more interesting is the overall architecture that could support much greater flexibility in the future. If dashboards gain the flexibility they need, and queries continue to be done in the conventional manner, query volumes will increase enormously. If it further is the case that they are upgraded in some near real-time manner, that’s another huge increase.
How huge? Well, I can make a case that it could be well over three orders of magnitude: Read more
| Categories: Aleri and Coral8, Analytic technologies, Business intelligence, Memory-centric data management, Streaming and complex event processing (CEP) | 3 Comments |
And then there is predictability
Coral8 at the time of a recent product release stated that it was improving the predictability of its queries. While this may sound like it has something to do with determinism, it doesn’t. Rather, it’s a matter of making what actually happens as a query result be more in line with what one would think will happen when one reads the query.
Coral8 CTO Mark Tsimelzon goes on to note:
But remember, we are really talking about a corner case — highly complex queries involving loops. We only had a couple of customers who were occasionally hitting queries that complex. The beauty of our SQL-based language is that the vast majority of queries, perhaps 99%, are very easy to understand, and their behavior is exactly what you’d expect based on your SQL experience.
| Categories: Aleri and Coral8, Memory-centric data management, Streaming and complex event processing (CEP) | Leave a Comment |
Applications for not-so-low-latency CEP
The highest-profile applications for complex event/stream processing are probably the ones that require super-low latency, especially in financial trading. However, as I already noted in writing about StreamBase and Truviso, there are plenty of other CEP apps with less extreme latency requirements.
Commonly, these are data reduction apps – i.e., there’s a gushing stream of inputs, and the CEP engine filters and “enhances” it, so that only a small, modified subset is sent forward. In other cases, disk-based systems could do the job perfectly well from a performance standpoint, but the pattern matching and filtering requirements are just a better fit for the CEP paradigm.
Read more
| Categories: Aleri and Coral8, IBM and DB2, Memory-centric data management, StreamBase, Streaming and complex event processing (CEP), Structured documents | 3 Comments |
Coral8 versus StreamBase
Besides talking about what Coral8 and StreamBase (and other CEP vendors) have in common, Mark Tsimelzon and I talked quite a bit about what he sees as some of the important differences. There were a lot, of course, but three in particular stood out.
1. Mark believes Coral8 has significantly lower latency than StreamBase. E.g., the Wombat/Coral8 combo achieves sub-millisecond latency, with Coral8 itself consuming less than a tenth of that. The best comparable figures from StreamBase that I currently know of are almost an order of magnitude slower.
Top-end speed aside, Mark believes that Coral8 is fundamentally better suited for complex queries and pattern recognition, while StreamBase works well with simpler queries. For example, his other performance claims notwithstanding, he concedes that StreamBase is at least comparable to Coral8 in its throughput for huge numbers of simple queries. (The number he mentioned was ½ million queries/second.) Indeed, while we barely talked about customer/marketing issues, Mark asserts that the companies’ respective customer bases reflect this complex/simple distinction.*
Read more
| Categories: Aleri and Coral8, Memory-centric data management, Progress, Apama, and DataDirect, StreamBase, Streaming and complex event processing (CEP) | 5 Comments |
The essence of CEP according to Coral8
Last week, I complained that my first briefing with Coral8 wasn’t very technical. Wednesday I had a call with Mark Tsimelzon, CTO and founder of Coral8, and he made up for that in spades. In this post I’ll cover some of his general comments. Others will touch on more Coral8-specific topics, and his view of the Coral8/StreamBase comparison.
As Mark describes it, the big difference between a DBMS – even an in-memory DBMS – and a complex event processing engine is this: CEP engines do instantaneous incremental processing. He commonly refers to this as registering queries and operators for incremental evaluation. For example, suppose you need to maintain the sum of some data stream over the past 10 minutes. Then each second (or other short unit of time), the system adds in all the values that arrived in the past second, and subtracts all those that arrived 600-601 seconds ago. Voila! The sum is incrementally updated.
Now, rolling sums may not sound very interesting – but where you have rolling sums, you trivially also have rolling averages (just divide the sum by the count) and rolling standard deviations (same idea, with some squares and square roots mixed in). Those, of course, are primitives in Coral8 too. Ditto rolling maxima and minima. Ditto rolling joins (which are updated a lot like materialized views).
Read more
| Categories: Aleri and Coral8, Memory-centric data management, Streaming and complex event processing (CEP) | 2 Comments |
Competitive claims in CEP
For the most part, the vendors I talk with in complex event/stream processing like and speak well of each other (most of the exceptions seem to involve StreamBase). Even so, there are a lot of interesting competitive claims and counterclaims in this market. Prior posts and comment threads have covered Apama/StreamBase jousting on the subjects of who has more business and how many financial data feeds StreamBase supports. Other areas that generate interesting sparks are performance, parallelism, and determinism. Read more
| Categories: Aleri and Coral8, Investment research and trading, Memory-centric data management, Progress, Apama, and DataDirect, StreamBase, Streaming and complex event processing (CEP) | 1 Comment |
The Coral8 story
Complex event/stream processing vendor Coral8 raised its hand and offered a briefing – non-technical, alas, but at least it was a start. Here are some of the highlights: Read more