Notes on CEP performance

I’ve been talking to CEP vendors on and off for a few years. So what I hear about performance is fairly patchwork. On the other hand, maybe 1-2+ year-old figures of per-core performance are still meaningful today. After all, Moore’s Law is being reflected more in core count than per-core performance, and it seems CEP vendors’ development efforts haven’t necessarily been concentrated on raw engine speed.

So anyway, what do you guys have to add to the following observations?

• Super-low-latency financial services industry tasks are often “embarrassingly parallel.” Thus, near-linear scale-out is common.
• That said, good parallelism seems fairly new in CEP engines (of course, CEP engines are fairly new themselves — for all I know, some have been parallel since inception).
• I’ve heard claims of up to 400,000 messages/second/core for simple queries or patterns.
• I’ve heard claims of 70,000 messages/core for not-so-simple queries or patterns, and probably higher than that depending on what the meaning of “simple” is.
• IBM just disclosed >15,000 messages/core on a pretty low-powered processor.
• I’ve heard that Coral8, Apama, and StreamBase rarely lost deals due to performance or throughput problems. I’ve heard that the same is not as true of Aleri.
• StreamBase proudly says it’s been fully multithreaded since academic research-project days.  For Apama multithreading is evidently a more recent feature. But does it matter much?

Followup on IBM System S/InfoSphere Streams

After posting about IBM’s System S/InfoSphere Streams CEP offering, I sent three followup questions over to Jeff Jones.  It seems simplest to just post the Q&A verbatim.

1.  Just how many processors or cores does it take to get those 5 million messages/sec through? A little birdie says 4,000 cores. Read more

IBM System S Streams, aka InfoSphere Streams, aka stream processing, aka “please don’t call it CEP”

IBM has hastily announced System S Streams, a product that was supposed to be called InfoSphere Streams and introduced only in 2010. Apparently, the rush is because senior management wanted to talk about it later this week, and perhaps also because it was implicitly baked into some of IBM’s advertising already. Scrambling ensued. Even so, Jeff Jones and team got to me fast, and briefed me — fairly non-technically, unfortunately, but otherwise how I like it, namely on a harmless embargo and without any NDAs. That’s more than can be said for my clients at Microsoft, who also introduced CEP this week, but I digress …

*Indeed, as I draft this post-Celtics-game, the embargo is already expired.

Marketing aside, IBM System S/InfoSphere Streams is indeed a CEP/stream processing engine + language (with an Eclipse-based development environment). Apparently, IBM’s thinks InfoSphere Streams (if that’s what it winds up being renamed to) is or will be differentiated from other CEP packages in:

• Scale-out. (That’s the one that appears to be real today. In fact, there’s a prototype running on Blue Gene.)
• Support for complex datatypes such as XML, text, voice, video, etc.
• Security and general industrial-strengthness.
  

Read more

Oracle’s hardware strategy

Larry Ellison stated clearly in an email interview with Reuters (links here and here) that Oracle intends to keep Sun’s hardware business and indeed intends to invest in the SPARC chip. Naturally, I have a few thoughts about this.

As Stephen O’Grady points out, Sun’s main strength lay in selling to the large enterprise market. Well, that’s Oracle’s overwhelming focus too. As I noted two years ago:

One Oracle response is to provide lots of add-on technologies for high-end customers, on the database and middle tiers alike. In app servers it’s done surprisingly well against BEA. It’s sold a lot of clustering. And it’s bought into and tried to popularize niche technologies like TimesTen and Tangosol’s.

This all makes perfect sense – it’s a great fit for Oracle’s best customers, and a way to get thousands of extra dollars per server from enterprises that may already have bought all-you-can-eat licenses to the Oracle DBMS. And being so sensible, it fits into the Clayton Christensen disruption story in two ways:

1. Oracle may be helpless against mid-tier competition, but it sure has the high-end core of its market locked up.

2. As one type of technology is commoditized, value is created in other parts of the technology stack.

Oracle’s ongoing acquisition spree in system software, application software, and now hardware just supports that story. MySQL, embedded Java, and so on may be welcome to Oracle as yet more opportunities to tap additional markets — but Oracle’s emphasis is and surely will remain on the large enterprise market.

The next notable point may be found in Larry’s key quote: Read more

Some DB2 highlights

I chatted with IBM Thursday, about recent and imminent releases of DB2 (9.5 through 9.7). Highlights included:

• DB2 is getting Oracle emulation, which I posted about separately.
• IBM says that it had >50 new DB2 data warehouse customers last year. I neglected to ask how many of these had been general-purpose DB2 customers all along.
• By “data warehouse customer” I mean a user for InfoSphere Warehouse, which previously was called DB2’s DPF (Data Partitioning Feature). Apparently, this includes both logical and physical partitioning. E.g., DB2 isn’t shared-nothing without this feature.
• IBM is proud of DB2’s compression, which it claims commonly reaches 70-80%. It calls this “industry-leading” in comparison to Oracle, SQL Server, and other general-purpose relational DBMS.
• DB2 compression’s overall effect on performance stems from a trade-off between I/O (lessened) and CPU burden (increased). For OLTP workloads, this is about a wash. For data warehousing workloads, IBM says 20% performance improvement from compression is average.
• DB2 now has its version of one of my favorite Oracle security features, called Label Based Access Control. A label-control feature can make it much easier to secure data on a row-by-row, value-by-value basis. The obvious big user is national intelligence, followed by financial services. IBM says the health care industry also has interest in LBAC.
• Also in the security area, IBM reworked DB2’s audit feature for 9.5
• I think what I heard in our discussion of DB2 virtualization is:
  • Increasingly, IBM is seeing production use of VMware, rather than just test/development.
  • IBM believes it is a much closer partner to VMware than Oracle or Microsoft is, because it’s not pushing its own competing technology.
  • Generally, virtualization is more important for OLTP workloads than data warehousing ones, because OLTP apps commonly only need part of the resources of a node while data warehousing often wants the whole node.
  • AIX data warehousing is an exception. I think this is because AIX equates to big SMP boxes, and virtualization lets you spread out the data warehousing processing across more nodes, with the usual parallel I/O benefits.
• When IBM talks of new autonomic/self-tuning features in DB2, they’re used mainly for databases under 1 terabyte in size. Indeed, the self-tuning feature set doesn’t work with InfoSphere Warehouse.
• Even with the self-tuning feature it sounds as if you need at least a couple of DBA hours per instance per week, on average.
• DB2 on Linux/Unix/Windows has introduced some enhanced workload management features analogous to those long found in mainframe DB2. For example, resource allocation rules can be scheduled by time. (The point of workload management is to allocate resources such as CPU or I/O among the simultaneous queries or other tasks that contend for them.) Workload management rules can have thresholds for amounts of resources consumed, after which the priority for a task can go up (”Get it over with!”) or down (”Stop hogging my system!”).

IBM’s Oracle emulation strategy reconsidered

I’ve now had a chance to talk with IBM about its recently-announced Oracle emulation strategy for DB2. (This is for DB2 9.7, which I gather has been quasi-announced in April, will be re-announced in May, and will be re-re-announced as being in general availability in June.)

Key points include:

• This really is more like Oracle emulation than it is transparency, a term I carelessly used before.
• IBM’s Oracle emulation effort is focused on two technological goals:
  • Making it easy for an Oracle application to be ported to DB2.
  • Making it easy for an Oracle developer to develop for DB2.
• The initial target market for DB2’s Oracle emulation is ISVs (Independent Software Vendors) much more than it is enterprises. IBM suggested there were a couple hundred early adopters, and those are primarily in the ISV area.

Because of Oracle’s market share, many ISVs focus on Oracle as the underlying database management system for their applications, whether or not they actually resell it along with their own software. IBM proposed three reasons why such ISVs might want to support DB2:

Read more

DBMS transparency layers never seem to sell well

A DBMS transparency layer, roughly speaking, is software that makes things that are written for one brand of database management system run unaltered on another.* These never seem to sell well. ANTs has failed in a couple of product strategies. EnterpriseDB’s Oracle compatibility only seems to have netted it a few sales, and only a small fraction of its total business. ParAccel’s and Dataupia’s transparency strategies have produced even less.

*The looseness in that definition highlights a key reason these technologies don’t sell well — it’s hard to be sure that what you’re buying will do a good job of running your particular apps.

This subject comes to mind for two reasons. One is that IBM seems to have licensed EnterpriseDB’s Oracle transparency layer for DB2. The other is that a natural upgrade path from MySQL to Oracle might be a MySQL transparency layer on top of an Oracle base.

Read more

Database implications if IBM acquires Sun

Reported or rumored merger discussions between IBM and Sun are generating huge amounts of discussion today (some links below). Here are some quick thoughts around the subject of how the IBM/Sun deal — if it happens — might affect the database management system industry.

Read more

IBM in the cloud

IBM is making DB2, Informix Dynamic Server, and other products available in the Amazon cloud.  The press release says test and development are free, while production will be charged at an hourly rate.  No doubt more price details will be forthcoming when the whole thing is fully in production.

Frankly, I’ve lost track of who all has some kind of cloud or SaaS offering now.  The list is at least Oracle, IBM, presumably Microsoft, MySQL (via Elastra, and also at almost every web host), PostgreSQL (ditto, more or less), EnterpriseDB, Kognitio, Vertica, Netezza, Aster Data.  No doubt I’m forgetting a bunch more.

Analytics’ role in a frightening economy

I chatted yesterday with the general business side (as opposed to the trading operation) of a household-name brokerage firm, one that’s in no immediate financial peril. It seems their #1 analytic-technology priority right now is changing planning from an annual to a monthly cycle.* That’s a smart idea. While it’s especially important in their business, larger enterprises of all kinds should consider following suit. Read more

Draft slides on how to select an analytic DBMS

I need to finalize an already-too-long slide deck on how to select an analytic DBMS by late Thursday night.  Anybody see something I’m overlooking, or just plain got wrong?

Edit: The slides have now been finalized.

DB2 engine coming soon for MySQL — but that’s DB2 on the iSeries

Chris Maxcer reports that a DB2 storage engine for MySQL is coming soon. But that’s specifically on the i Series — i.e., the heirs of the AS/400 and before that System 38 product lines.   While those are arguably the best systems IBM ever produced, it’s still a non-event for most of the IT market. DB2 isn’t DB2 isn’t DB2 …

Gartner’s 2009 Magic Quadrant for Business Intelligence

A few days ago I tore into the Gartner Magic Quadrant for Data Warehouse DBMS.  Well, the 2009 Gartner Magic Quadrant for Business Intelligence Platforms is out too. (Link here.  Last year’s here. Hat tip for both to Doug Henschen.)  Unlike the data warehouse MQ, Gartner’s BI MQ clusters its “Leaders” together tightly. But while less bold, the Business Intelligence Magic Quadrant’s claims are just as questionable as those in data warehousing.

Of course, some parts do make sense.  E.g.: Read more

Gartner’s 2008 data warehouse database management system Magic Quadrant is out

Gartner’s annual Magic Quadrant for data warehouse DBMS is out.  Thankfully, vendors don’t seem to be taking it as seriously as usual, so I didn’t immediately hear about.  (I finally noticed it in a Greenplum pay-per-click ad.)  Links to Gartner MQs tend to come and go, but as of now here are two working links to the 2008 Gartner Data Warehouse Database Management System MQ.  My posts on the 2007 and 2006 MQs have also been updated with working links. Read more

MySQL is being used in an IBM Lotus appliance

Apparently, IBM is rolling out an appliance for small businesses. MySQL is under the covers. The appliance won’t have a keyboard or monitor, so there won’t be a lot of database administration going on.

Before Solid and solidDB were acquired by IBM, one of the things Solid was proudest of was some embedded apps in which solidDB ran for years in boxes without keyboards or monitors.

I still think it’s a pity that IBM isn’t using solidDB as broadly as the technology deserves. Even so, this is a nice endorsement of MySQL for reliable zero-DBA mid-range use.

Big scientific databases need to be stored somehow

A year ago, Mike Stonebraker observed that conventional DBMS don’t necessarily do a great job on scientific data, and further pointed out that different kinds of science might call for different data access methods. Even so, some of the largest databases around are scientific ones, and they have to be managed somehow. For example:

• Microsoft just put out an overwrought press release. The substance seems to be that Pan-STARRS — a Jim Gray legacy also discussed in an August, 2008 Computerworld article — is adding 1.4 terabytes of image data per night, and one not so new database adds 15 terabytes per year of some kind of computer simulation output used to analyze protein folding. Both run on SQL Server, of course.
• Kognitio has an astronomical database too, at Cambridge University, adding 1/2 a terabyte of data per night.
• Oracle is used for a McGill University proteonomics database called CellMapBase. A figure of 50 terabytes of “mass storage” is included, which doesn’t include tape backup and so on.
• The Large Hadron Collider, once it actually starts functioning, is projected to generate 15 petabytes of data annually, which will be initially stored on tape and then distributed to various computing centers around the world.
• Netezza is proud of its ability to serve images and the like quickly, although off the top of my head I’m not thinking of a major customer it has in that area. (But then, if you just sell software, your academic discount can approach 100%; but if like Netezza you have an actual cost of goods sold, that’s not as appealing an option.)

Long-term, I imagine that the most suitable DBMS for these purposes will be MPP systems with strong datatype extensibility — e.g., DB2, PostgreSQL-based Greenplum, PostgreSQL-based Aster nCluster, or maybe Oracle.

Schema flexibility and XML data management

Conor O’Mahony, marketing manager for IBM’s DB2 pureXML talks a lot about one of my favorite hobbyhorses — schema flexibility — as a reason to use an XML data model. In a number of industries he sees use cases based around ongoing change in the information being managed:

• Tax authorities change their rules and forms every year, but don’t want to do total rewrites of their electronic submission and processing software.
• The financial services industry keeps inventing new products, which don’t just have different terms and conditions, but may also have different kinds of terms and conditions.
• The same, to some extent, goes for the travel industry, which also keeps adding different and different kinds of destinations.
• The energy industry keeps adding new kinds of highly complex equipment it has to manage.

Conor also thinks market evidence shows that XML’s schema flexibility is important for data interchange.

Read more

Vertical market XML standards

Tracking the alphabet soup of vertical market XML standards is hard. So as a starting point, I’m splitting a list I got from IBM into a standalone post.

Among the most important or successful IBM pureXML-supported standards, in terms of downloads and other evidence of customer interest, are:

Read more

Overview of IBM DB2 pureXML

On August 29, I had a great call with IBM about DB2 pureXML (most of the IBM side of the talking was done by Conor O’Mahony and Qi Jin). I’m finally getting around to writing it up now. (The world of tabular data warehousing has kept me just a wee bit busy …)

As I write it, I see there are a considerable number of holes, but that’s the way it seems to go when researching XML storage. I’m also writing up a September call from which I finally figured out (I think) the essence of how MarkLogic Server works – but only after five months of trying. It turns out that MarkLogic works rather differently from DB2 pureXML. Not coincidentally, IBM and Mark Logic focus on rather different use cases for native XML storage.

What I understand so far about the basic DB2 pureXML architecture goes like this:

Read more

Top DBMS on Linux

I was looking up George Crump’s blogs in connection with his recent post on SSDs, and I stumbled upon one that outlines at great length what features Linux backup systems should have. I won’t claim to have read it word for word, but what did catch my eye were a couple of comments on DBMS market share, which boiled down to:

1. Oracle
2. MySQL
3. PostgreSQL

Read more

More data on data warehouse sizes and issues

I spoke today with Paul Barth and Randy Bean of consultancy NewVantage Partners. The core of NewVantage’s business seems to be helping large enterprises (especially financial services) with their data warehouse strategies. Takeaways — none of which should shock regular readers of DBMS2 — included:

• Administrative cost and difficulty are often the single biggest issue in selecting analytic DBMS products.
• Oracle hits a wall around 10 terabytes of user data. The one customer NewVantage can think of with an Oracle data warehouse over 10 terabytes is fleeing Oracle for Netezza.
• NewVantage says that very specialized data warehouses on Oracle could conceivably be larger than that.
• NewVantage does have a customer on DB2/UDB in the 30-40 terabyte range. That customer does a lot of careful tuning to make it work.
• About 15% of NewVantage’s customers use Netezza. Few if any use newer analytic DBMS (but I got the sense more will soon). The rest rely on “traditional” DBMS, a group that includes Teradata.

Microsoft is buying DATAllegro

I’ve long argued that:

• Oracle and Microsoft are doomed in the data warehouse market unless they acquire MPP/shared-nothing data warehouse DBMS and/or data warehouse appliances.
• DATAllegro is the ideal acquisition for either of them.

Microsoft has now validated my claim by agreeing to buy DATAllegro. As you probably know, we’ve been covering DATAllegro extensively, as per the links listed below.

Basic deal highlights include:

Read more

Who is doing what in XML data management these days?

A comment thread to a post on a different subject has opened up a discussion of XML storage. Frankly, I haven’t kept up with my briefings on the subject, in part because XML support hasn’t proved to be very important yet to the big DBMS vendors, somewhat to my surprise. When last I looked, the situation wasn’t much different from what it was back in November, 2005. Unless I’ve missed something (and please tell me if I have!), here’s what’s going on: Read more

ANTs bails out of the DBMS market

ANTs Data Server — i.e., the ANTs DBMS — has been sold off to a company called 4Js. It is now to be called Genero DB. Actually, 4Js has been selling or working on a version of the product called Genero DB since 2006, specifically an Informix-compatible one.

I’m not totally clear on why an Informix-compatible DBMS is needed in a world that already has Informix SE, but maybe IBM is overcharging for maintenance even on the low-end version of the product.

Meanwhile, ANTs, which had originally tried to get enterprises to migrate away from Oracle, is now focused on middleware called the ANTs Compatibility Server to help them migrate to Oracle, specifically/initially from Sybase.

Truviso and EnterpriseDB blend event processing with ordinary database management

Truviso and EnterpriseDB announced today that there’s a Truviso “blade” for Postgres Plus. By email, EnterpriseDB Bob Zurek endorsed my tentative summary of what this means technically, namely:

• There’s data being managed transactionally by EnterpriseDB.

• Truviso’s DML has all along included ways to talk to a persistent Postgres data store.

• If, in addition, one wants to do stream processing things on the same data, that’s now possible, using Truviso’s usual DML.

Read more

Next Page →

Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

• DBMS2 covers database management, analytics, and related technologies.
• Text Technologies covers text mining, search, and social software.
• Strategic Messaging analyzes marketing and messaging strategy.
• The Monash Report examines technology and public policy issues.
• Software Memories recounts the history of the software industry.

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.

• Recent posts

  • User data vs. raw disk space as a marketing metric
  • The TPC-H schema
  • Notes on columnar/TPC-H compression
  • Storage humor
  • NoSQL?

• Categories

  • About this blog
  • Analytic technologies
    • Business intelligence
    • Data mart outsourcing
    • Data warehousing
    • MOLAP
  • Application areas
    • Games and virtual worlds
    • Investment research and trading
    • Scientific research
    • Telecommunications
    • Web analytics
  • Buying processes
    • Benchmarks and POCs
  • Companies and products
    • 1010data
    • Ab Initio Software
    • Aleri and Coral8
    • Alpha Five
    • Amazon and its cloud
    • ANTs Software
    • Aster Data
    • Business Objects
    • Calpont
    • Cast Iron Systems
    • Clearpace
    • Cloudera
    • Cogito and 7 Degrees
    • Cognos
    • CouchDB
    • DATAllegro
    • Dataupia
    • Elastra
    • EMC
    • EnterpriseDB and Postgres Plus
    • Exasol
    • Expressor
    • FileMaker
    • Google
    • Greenplum
    • Hadoop
    • HP and Neoview
    • IBM and DB2
      • pureXML
    • illuminate Solutions
    • Infobright
    • Informatica
    • Information Builders
    • Inforsense
    • Ingres
    • Intel
    • Intersystems and Cache'
    • Jaspersoft
    • Kalido
    • Kickfire
    • Kognitio
    • Mark Logic
    • McObject
    • Microsoft and SQL*Server
    • MySQL
    • Netezza
    • Oracle
      • Exadata
    • Oracle TimesTen
    • ParAccel
    • Pentaho
    • Pervasive Software
    • PostgreSQL
    • Progress, Apama, and DataDirect
    • QlikTech and QlikView
    • SAND Technology
    • SAP AG
    • SAS Institute
    • ScaleDB
    • Software AG
    • solidDB
    • StreamBase
    • Sybase
    • Tableau Software
    • Talend
    • Teradata
    • Tokutek
    • Truviso
    • Vertica Systems
    • VoltDB and H-Store
  • Data integration and middleware
    • Application servers
    • EAI, EII, ETL, ELT, ETLT
  • Data types
    • GIS and geospatial
    • Native XML
    • Object
    • RDF and graphs
    • Text
  • DBMS product categories
    • Archiving and information preservation
    • Data warehouse appliances
    • Mid-range
    • OLTP
    • Open source
  • Emulation, transparency, portability
  • Fun stuff
    • Humor
  • Market share
  • Memory-centric data management
    • Cache
    • Complex event processing (CEP)
    • In-memory DBMS
  • Michael Stonebraker
  • Parallelization
    • MapReduce
  • Presentations
  • Pricing
  • Software as a Service (SaaS)
    • Cloud computing
  • Specific users
    • eBay
    • Facebook and Cassandra
    • Fox and MySpace
    • TEOCO
    • Yahoo
  • Storage
    • Solid-state memory
  • Theory and architecture
    • Columnar database management
    • Data models and architecture
    • Database compression
    • Database diversity
    • Petabyte-scale data management
  • TransRelational
  • Uncategorized

• Date archives

  Select Month July 2009  (6) June 2009  (25) May 2009  (19) April 2009  (30) March 2009  (23) February 2009  (29) January 2009  (18) December 2008  (10) November 2008  (13) October 2008  (29) September 2008  (60) August 2008  (28) July 2008  (23) June 2008  (7) May 2008  (16) April 2008  (18) March 2008  (22) February 2008  (25) January 2008  (35) December 2007  (18) November 2007  (12) October 2007  (31) September 2007  (16) August 2007  (11) July 2007  (9) June 2007  (16) May 2007  (6) April 2007  (18) March 2007  (17) February 2007  (8) January 2007  (13) December 2006  (1) November 2006  (1) October 2006  (7) September 2006  (17) August 2006  (6) July 2006  (12) June 2006  (1) May 2006  (7) April 2006  (7) March 2006  (2) February 2006  (4) January 2006  (13) December 2005  (9) November 2005  (9) October 2005  (10) September 2005  (1) August 2005  (6)

• Links

  • Monash Research
  • Webcasts
  • White Papers

• Admin

  • Log in

• Home
• About
• Contact
• Feeds