• The upshot of my what to call CEP thread in August was that “streaming” and “event processing” are not the same concept, but it so happens that they have the most traction where they intersect. That said, I both observe and endorse an apparent shift from “event” to “stream” as the core of the terminology, in a reversal of my opinion of several years ago.
• IBM continues to throw a lot of resources at its System S/ InfoSphere Streams product, but I haven’t heard yet of much marketplace success. That said, I believe IBM is still pretty serious about Streams, as one would expect from an effort whose code name so cheekily references System R. In particular, Streams shows up prominently on IBM’s top-level analytic architecture slide.
• Sybase recently released its ESP (Event Stream Processor) 5.0, which it says is the full merger of the Aleri and Coral8 predecessors. You can still get Sybase ESP without buying into the full Sybase RAP stack, and Sybase has no plans to change that.
• Sybase has discontinued all the business intelligence types of products Aleri and Coral8 were developing. Rather, Sybase is OEMing Panopticon, which it reports has been well received. Other than the discontinuation of the BI efforts, there seem to be few Aleri or Coral8 features missing from the merged Sybase ESP product.
• Truviso continues to be out of the picture.
• I have more to say about StreamBase separately.
• I have more to say about Sybase and IBM, which I’ll get to when I can.
• I have nothing new on Progress Apama. I also know little about any of the open source efforts.

Meanwhile, if you want to see technically nitty-gritty posts about the CEP/streaming area, you may want to look at my CEP/streaming coverage circa 2007-9, based on conversations with (among others) Mike Stonebraker, John Bates, and Mark Tsimelzon.

• Log-stream machine-generated data, when what you’re looking at — at least initially — is the entire output of verbose logging systems.
• Remote machine-generated data.

Here’s what I’m thinking of for the second category. I rather frequently hear of cases in which data is generated by large numbers of remote machines, which occasionally send messages home. For example: 

• I heard yesterday about a case with 10s of millions of machines, phoning home every 5 minutes, and another case with 10s of 1000s calling in every 5 seconds, both of them sending data initially to MySQL. (Application details weren’t given.)
• I heard not long ago about a set-top box case that the vendor hoped would also grow to 10s of millions of machines, which I guessed might send a small number of messages per hour each.
• I also heard recently about a remote security monitoring case whose data was destined for (probably) Netezza, although in that case I’m not sure about the “occasionally” aspect of the communication.
• The last time I visited Splunk, I got the sense that energy-sensor use cases (especially in the electric grid) had finally emerged. I believe these sensors are periodic message senders — they wake up, take their temperature (figuratively or literally as the case may be), send a message, snooze, repeat.
• I would guess that the energy use cases Infobright talked about in 2009 were of a similar kind.
• An April, 2010 comment on the post linked above talks about many kinds of sensor data.
• Back in 2007, Coral8 talked of a truck phone-home use case (on-board GPS data and also, e.g., refrigeration level, sending messages once a minute or so). Truviso seemed to have one similar deal before one of its frequent changes in strategic direction, and not coincidentally cites UPS as an investor.
• In principle, there are a lot of RFID use cases out there, even if I rarely seem to hear of any. (That would be a shorter “phone call” home than most of the other examples, of course, but might be otherwise technically similar.)

Many technologies can be used to collect and manage remote machine-generated data, but a few common points are worth nothing.

• If a device takes the trouble to send a message across a wide-area network, that message may be somewhat more valuable than the average piece of log-vomit. Perhaps such information doesn’t need to be stored in the cheapest possible way.
• Similarly, a message that is sent occasionally over time, or upon a specified event, may be more structured than a random log entry. Perhaps such data is suitable for sending straight to a relational database.
• If there’s no central place the data originates, there may also be no favored place for the data to end up. It may make great sense to collect and analyze remote machine-generated data in the cloud. (Exceptions may of course arise if you want to use the data in connection with other information, and you hence want to bring it to that other information’s location.)
• In a number of use cases, the whole point is to identify anomalies, and respond to them rapidly. I.e., remote machine-generated data use cases commonly raise challenges in low-latency integration of short-request and analytic processing.

Meanwhile, Truviso — which never got much market traction in the first place — was in hiding; Roman Bukary never did keep his promise to brief me on the company’s new and improved strategy. Then Truviso had yet another management change, amidst rumors that it was repositioning away from CEP. As per a press release Truviso emailed today, that’s now official, with Truviso’s main business being something to do with web analytics.

• StreamBase once planned to conquer the world, making an impact as big as database management’s. Now it has retreated into niche markets.
• Progress Software, a decent-sized company, put a large fraction of its energy into Apama. Little has happened outside the financial service sector.
• Coral8 has some great-sounding ideas. But Coral8 now has merged into Aleri, basically a financial-markets specialist.
• Mike Franklin says some ambitious things on behalf of Truviso, but I haven’t noticed much traction there either.

CEP’s penetration outside of its classical markets isn’t quite zero. Customers include several transportation companies (various vendors), Sallie Mae (Coral8), a game vendor or two (StreamBase, if I recall correctly), Verizon (Aleri, I think), and more. But I just wrote that list from memory — based mainly on not-so-recent deals — and a quick tour of the vendors’ web sites hasn’t turned up much I overlooked. (Truviso does have a recent deal with Technorati, but that’s not exactly a blue chip customer these days.)

So far as I can tell, this is a new version of a repeated story. A clever alternative to relational DBMS was invented. It proved superior in some specific applications and vertical markets. It failed to achieve much broader adoption. Initial high hopes got dashed, companies failed to grow rapidly, and shareholders grew tired.

So where will things go from here? My best guesses include:

• The financial trading market isn’t going away for CEP. Super-low-latency is really needed there.
• As much as I love the idea of CEP-infused BI, it will be adopted only at the rate broader-based BI vendors can support.
• A few niches will generate some business for CEP in data reduction. Leading candidates are the ones where there’s been a little traction to date — national intelligence, transportation, web analytics, and so on.
• Sadly, that’s about it.

]]> http://www.dbms2.com/2009/03/09/independent-cep-vendors-continue-to-flounder/feed/ 9 http://www.dbms2.com/2008/04/29/truviso-and-enterprisedb-blend-event-processing-with-ordinary-database-management/ http://www.dbms2.com/2008/04/29/truviso-and-enterprisedb-blend-event-processing-with-ordinary-database-management/#comments Wed, 30 Apr 2008 03:52:45 +0000 Curt Monash http://www.dbms2.com/?p=413 Truviso and EnterpriseDB announced today that there’s a Truviso “blade” for Postgres Plus. By email, EnterpriseDB Bob Zurek endorsed my tentative summary of what this means technically, namely:

• There’s data being managed transactionally by EnterpriseDB.

• Truviso’s DML has all along included ways to talk to a persistent Postgres data store.

• If, in addition, one wants to do stream processing things on the same data, that’s now possible, using Truviso’s usual DML.

Note: Extended-relational DBMS like Postgres, Oracle, DB2, and Informix/Illustra have long offered the ability to add blades/cartridges. It’s easy to understand what these do when they simply add native management for a new datatype, and extend the parser, optimizer, and data access methods accordingly. But blades are used in other ways as well, and I’ve always found that somewhat confusing. A little bit of that appears to be going on in this case.

Bob added that there have been a lot of inquiries about the announcement today, without specifying from whom. Truviso marketing chief Roman Bukary, late of SAP, sent over some generic use cases, which pretty much boil down to my first two bullet points above. (More precisely, they agree if you replace “transactionally” with “persistently”; Roman also foresees data warehousing uses.)

I like this announcement. With one probable exception, it’s a good fit for every major use of event processing; the exception is super-low-latency apps, where no extraneous overhead is tolerable. (Those are found mainly in algorithmic trading, but could arise in security and network management as well.) But then, Truviso is being positioned away from its initial currency trading focus anyway.

Super-low-latency aside, the other big current use case for event processing is data reduction. I.e., you have a lot of incoming data – e.g., via satellite telemetry or intelligence intercepts or network monitoring sensors, or monitoring character movement in an MMO (Massively Multiplayer Online) game. You try to grab all the “interesting” stuff, while disregarding or even throwing away the rest. But the “throwing away” part is a little worrisome. So if instead you can seamlessly persist everything, even for a short period of time (e.g., measured in days), that’s goodness. Even if you can’t keep it all even for a short while – well, if the point of data reduction is to retain only a fraction of the incoming data, this scheme could make it easier to persist the keepers.

Another current use case for event processing is rules engines. Progress Apama has a rules paradigm all the way down, while Coral8 tells happily of a customer who uses event processing for all kinds of rules-based real-time CRM. But the Coral8 example is closely integrated with conventional persistent data stores, and the same is likely for other similar applications. Business activity monitoring (BAM) would be a special case of this.

As you know, my ultimate dream for business intelligence/analytic uses of event processing goes beyond BAM. I think many individuals in an enterprise should each track many different (but related) KPIs (Key Performance Indicators). Current query loads for reporting, dashboards, ad hoc query, etc. could easily go up by 2-3 orders of magnitude. When that happens, you want to consider different ways of doing things, specifically memory-centric ones. Normal memory-centric data processing might get the job done, but I have a suspicion that the right architecture will wind up looking a lot like event processing.

Once again, that’s a use for event processing that naturally integrates tightly with a persistent database.

Related links:

• Truviso’s version of the announcement

• An earlier press release declaring Truviso’s love for PostgreSQL

But my biggest takeaway from this pair of calls was that Coral8 and Truviso are penetrating general BI. It’s not a huge trend yet — maybe 20-30 customers total between them. Most of those are Coral8′s, and specifically focused on clickstream analysis and the like. Truviso’s apps are still double-deep secret stealth mysteries, but Roman Bukary promises they’ll be unveiled in a couple of months or so.

I gave both Roman and John Morell of Truviso a choice between three types of use:

1. Feeding operational apps directly
2. Feeding a dashboard
3. Feeding a data warehouse for subsequent analytics (reporting, data mining, dashboard, whatever)

Both indicated that their companies were active in the first choice, feeding operational apps. Roman said Truviso is also driving dashboards, but isn’t doing a lot of what amounts to ETL for warehouses. Coral8, meanwhile, is feeding warehouses more than directly driving dashboards.

Meanwhile, I continue to think that event processing technology is the future of dashboards.

Please subscribe to our feed!

Apparently, Truviso convinced him that other CEP firms execute one query at a time, while Truviso executes a bunch of queries at once. Well, the latter part of that is presumably true, but it’s hardly the big differentiatior for Truviso Robin would have one believe. That’s what everybody else — StreamBase, Coral8, Progress Apama, et al. — do too. I wouldn’t be surprised if Truviso had a somewhat different architecture for doing it (each vendor describes its approach in rather different language), or even if this were a particular focus and strongpoint of theirs. But fundamentally, all the CEP vendors are doing the same thing.

• The current need is for trades to be completed in 10 milliseconds, give or take; my 30 milliseconds figure was old news. Database lookups should be no more than 1/10 of that. I wasn’t too clear, actually, on how many lookups had to occur in that 1 millisecond or so, but no matter for now. The main point is that disk lookup will not get the job done.
• Because of that, Mike challenges Truviso’s assertion that close integration with disk-based DBMS matters. (Truviso credibly claims that its integration into PostgreSQL is tighter than StreamBase has with Sleepycat.) If you need to use data from disk, you’d better suck it all into memory in advance. For this reason and others, apps commonly need to preserve “state” totaling up to a few gigabytes. StreamBase also has ODBC/JDBC access to external databases, but that takes 30 milliseconds or so, which is too slow for financial apps. (In retrospect, I should have pushed back more about integration w/ disk-based DBMS in non-financial apps that aren’t so demanding about latency.)
• Interestingly, standard stock ticker feeds such as Reuters have latency up to 100 milliseconds or so. Hence, large investors take raw feeds from stock exchanges and so on.
• Stream processor architectures are radically different from those of conventional DBMS, even beyond the obvious ways. E.g., there’s no time to put tasks on a queue. Each StreamBase step, if at all possible, calls its successor directly. And filters (i.e., queries or query parts) are compiled straight down into machine code, whereas conventional DBMS usually stop at p-code and then interpret from there.
• The performance wars, in Mike’s opinion, are fought largely – but not entirely — around which features have great performance. Sophistication of windowing is one example, presumably since any way to slice data into “windows” is apt to be an uncomfortable approximation to what you really want to know.
• Another is stream disorder (possibly an infelicitous phrase if your audience is into toilet humor). For starters, one needs to be able to deal with data that arrives with its timestamps out of order. For some apps (sensor networks perhaps more than financial ones) one even needs to handle cases where some data doesn’t arrive at all, at least not in an acceptable timeframe.
• Incredibly important is intrastream pattern recognition – e.g., self-joins of streams.
• While Mike concedes that StreamSQL is incomplete as SQL goes, he claims nobody cares about the missing features (e.g., nested queries). Take that, Truviso! He also thinks there’s going to be an agreed-upon de facto standard for stream extensions to SQL. That said, precedent suggests to me that this standard will be in the form of extensions, ala SQL/MM and so on. Thus – unless and until I’m proven wrong on that — the idea of an emerging standard is not by itself an argument for lesser vs. richer base SQL.
• StreamBase and Truviso alike make a big deal about having super-high performance “replaying” sequential streams for backtesting. Presumably, this means that you suck historical data in in a batch, then check query results as if it had streamed in. I don’t know how the performance on this compares to what happens when the data comes in live.

Next up, I’ve been assured, is a talk with Progress Software’s Apama division.

Both companies’ core product is a memory-centric SQL engine designed to execute queries without ever writing data to disk. Of course, they both have persistence stories too — Truviso by being tightly integrated into open-source PostgreSQL, StreamBase more via “yeah, we can hand the data off to a conventional DBMS.” But the basic idea is to route data through a whole lot of different in-memory filters, to see what queries it satisfies, rather than executing many queries in sequence against disk-based data.

The basic paradigm for filtering – certainly in Truviso’s case, and I think in StreamBase’s as well – is from the columnar/bitmap school. A record is a vector or a set of vectors. A query or filter is a vector or set of vectors. So take some dot products, see where you are, and you’ll know if the query is satisfied.

Not coincidentally, I would think, this is very similar to the approach taken by various memory-centric BI vendors, such as QlikTech, or SAP in its BI Accelerator. It’s also the approach taken by text indexers. Indeed, the first product I ever heard of of this kind was actually in the text area, from Verity, a decade ago, notwithstanding that Verity had a very small team for sophisticated DBMS types of things. (The whole kernel group was 6-7 people.) And also perhaps not coincidentally, StreamBase reports that despite a lack of text processing primitives, a lot of the use of their technology is for text filtering, I presume in national-security kinds of applications.

The core market for this stuff is financial trading, where the rule of thumb is that a complete query, decision, and transaction has to be done in 30 milliseconds, and database lookup gets to use less than 10% of that time. (Hence the memory-centric requirement; there’s simply no way to search a disk usefully in less than 4-7 milliseconds.) That’s also where Progress Apama (which has a rules-based approach to the same problems) is focused.

The two other obvious and traditional markets for memory-centric technology are national security and telecommunications. (E.g., there’s no prize for guessing which three industries are called out on StreamBase’s website.) But it goes further than that. For example,

• MMORPGs (Massively MultiPlayer Online Role-Playing Games). I forget how many details StreamBase has disclosed by now, but they have something going in that space. By the way, I also have a research project ongoing about the technology of MMORPGs, for an eventual Network World column. I love my work!
• Online travel/reservations demand management. StreamBase is active there too.
• Logistics. UPS is an investor in Truviso.
• General real-time BI.

Adopting this technology is easier than one might at first think. StreamBase has a nice Eclipse-based development tool. Truviso has some packaged BI/visualization. StreamBase notes that one use of its technology is preprocessing input streams for existing or conventional applications. Truviso claims to make ETL easier as well, although I confess to so far only having seen their smoke on that subject, and not yet detected the associated fire.

One point StreamBase noted is that this preprocessing is not just for speed/latency. Especially in the intelligence business, the raw data is huge. Vast amounts of data reduction are an absolute requirement. As we go to civilian deployments of widespread sensor technology – RFID, GPS/presence, whatever – similar needs may well arise.

I could and probably at some point will say a lot more on these subjects. There are technical details (e.g., Truviso fondly claims to have a more complete product, despite being a much smaller company that’s been in business for a much shorter length of time, apparently because it bundles in more SQL of the nature suited for disk-centric systems). There’s how this all fits into my thesis on the disruption/reinvention of BI, and what’s still lacking – some more personal alert/KPI management, guys, if you please! And so on. Please stay tuned.

And please let me know what you think based on what you know today.