• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

*Links to Gartner Magic Quadrants commonly break, but that one worked at the time of this posting.

• Gartner thinks that data warehouse appliances are on the rise, due to their simplicity.
• Gartner correctly says that Teradata has been a data warehouse appliance vendor from the getgo.
• Gartner characterizes IBM as being an appliance vendor as well.
• Gartner suggests that HP is having trouble living up to its technical promises for Neoview.
• Gartner further suggests — no surprise here — that HP Neoview has had very few new customers past its initial wave.
• Gartner notes IBM’s difficulties in selling data warehouse installations of DB2, despite what on paper is great-sounding technology.
• Gartner says — also no surprise — that illuminate “has seen little success in North America since opening its first office in the U.S. over two years ago.”
• Ingres has evidently gotten a few BI-centric “appliance” deals, e.g. with Jaspersoft. But basically Ingres isn’t doing well in data warehousing.
• Gartner does say Ingres has “the strongest open-source DBMS offering for data warehousing.” Being very literal about “open source,” that’s a defensible claim — but it’s pretty irrelevant in a world where Greenplum Single-Node Edition can be had for free. It also waves away all the data mart use cases in which Infobright Community Edition shines.
• Gartner says that Netezza is working out as a “complex workload” enterprise data warehouse provider, according to reference checks, in addition to its established success in data mart scenarios.
• Gartner says Oracle’s offering has finally become “accepted” in the market for databases >50 TB. I guess I can live with that fairly weak claim, but I wouldn’t go much further than that.
• Gartner asserts that, unlike software-only Oracle, Oracle Exadata isn’t significantly harder to administer than “other mixed OLTP/OLAP DBMS vendors,” because Exadata is fast enough you don’t need to jump through all those hoops any more to get tolerable performance. The money quote is “one reference reported reducing the number of indexes by a factor of 100 to fewer than five.” Note, however, that Gartner does not seem to assert that Exadata’s ease of use rivals that of the newer analytic DBMS specialists.
• Gartner confirms Oracle’s reluctance to do onsite Exadata POCs, but says it is not absolute. This is roughly compatible with what I’m hearing elsewhere, and indeed with Oracle own claims to be ramping up availability of Exadata POC hardware.
• Gartner’s criteria for inclusion include at least 10 different organizations having a product “in production.” Thus, the big surprise was ParAccel being included. The money quote there is “With approximately 20 customers in the pharmaceutical, retail, financial and media/advertising analytics sectors, ParAccel has a good reference base.” That assessment is difficult to reconcile with other information, but I’ve been told Gartner is sticking to its guns. That assessment would be even harder to believe if those 20 references were all alleged to be true production customers.
• Gartner notes that you basically can’t run a 1 TB+ MySQL data warehouse without sharding. (Of course, Infobright has an alternative, and up to a small number of terabytes so does Kickfire.)
• Gartner reports that at least some customers are pleased with Sybase IQ’s mixed workload/enterprise data warehouse capabilities.
• Gartner correctly notes that Oracle Exadata is a price-competition challenge for Teradata.
• Gartner notes that 20% of Vertica’s customers are outside the US. While not shocking, that’s more than I realized.
• Gartner notes something I don’t think I’ve posted yet, which is that Vertica has a customer with 300 TB of data. (The identity is a deep dark secret, but if I told you you probably wouldn’t recognize the name anyway.)

As does any such piece, the Gartner Data Warehouse DBMS Magic Quadrant also has outright errors.  For example:

• Aster Data isn’t really “the newest entrant to the DBMS data warehouse world.”
• Aster’s SQL/MapReduce was not new in Release 4.0.
• Greenplum isn’t yet pushing down code to the storage tier.
• I’m not sure what kind of database-tier parallelism Gartner is claiming is new in Oracle in 11g Release 2 — but I doubt it’s really new. Rather, what Oracle has done recently is make parallelism less administratively cumbersome.
• Vertica wasn’t really the first DBMS in the cloud. At most it was the first pure-play analytic DBMS to get there.

• Xkoto sells Gridscale, a clustering server for DB2 and, more recently, MS SQL Server.
• Xkoto Gridscale runs on a separate box, between the application and the database servers. This box is typically smaller and cheaper than the database server boxes.
• Xkoto most typically sells Gridscale into environments where there already are three database servers — one to do work, one for hot standby, and one for remote disaster recovery.
• In such environments, Gridscale’s big benefit is that you can distribute the query workload among all three servers. Xkoto believes this big performance increase is the reason customers don’t get much past 3 database servers under Xkoto (they didn’t seem quite sure as to whether the all-time record was 4 or 5). Note that even if a remote server is a little too far away for OLTP query response, it can work fine for reporting.
• Of course, if you don’t already have high/”continuous” availability and/or disaster recovery, then Xkoto would say those are core benefits of Gridscale as well.
• Gridscale sends transactions (or just SQL statements?) to all servers in the cluster. Once any of them responds affirmatively, that update is reflected in queries. Gridscale maintains a small query log to make sure it gets the other database copies in sync. It also tries to make sure that queries always go to the most current copy of the database. (I didn’t ask what happens if Server A executes Transaction T but not U, while Server B executes Transaction U and not T — but that does seem like something of an edge case.).
• Xkoto spun out of Halcyon Monitoring in 2006, starting with DB2 support. Microsoft SQL Server support was introduced in 2008.
• Xkoto likes its partnerships with IBM and Microsoft. For example, IBM provides Level 1 and 2 support for Gridscale itself. Due in large part to this partnership strategy, Xkoto says it has no plans to support DBMS beyond DB2 and SQL Server.
• Instead, Xkoto is pursuing partnerships with large application vendors and so on. (The figure “about 10″ was mentioned.) I gather the idea is to make sure that neither the application support folks nor the app itself freak out from the fact that the app isn’t exactly talking to the DBMS any more.
• Xkoto has done lab tests suggesting Gridscale offers near-linear scalability (in terms of SQL Server database throughput) on a query-only workload up to 10 servers.
• I gather that Xkoto and IBM have demos suggesting it’s a fine idea to have your disaster recovery server be in the Amazon cloud, but they haven’t yet made any sales based on that — er, based on that premise.
• Gridscale pricing is measured in the same metrics as DB2 or SQL Server pricing, and in each case is around 1/3 what database pricing would be on the same box (I’m guessing that’s for enterprise additions without add-ons, but I didn’t probe). Specifically, Gridscale charges $12K per 100 PVUs for the DB2 edition, and $12K per socket for running with Microsoft SQL Server.
• Gridscale typically runs on smaller boxes than the databases it talks to.
• Xkoto has about 35 revenue-recognized customers. Most are on DB2, the first environment Gridscale supported.
• Average Gridscale selling prices are $180K on DB2, and $40-50K in the early going for SQL Server.
• Xkoto has about 40 full-time employees, with engineering in Toronto and business operations in Waltham.

1) good buy for IBM? why or why not?

Yes. The integration of predictive analytics with other analytic or operational technologies is still ahead of us, so there was a lot of value to be gained from SPSS beyond what it had standalone.  (That said, I haven’t actually looked at the numbers, so I have no comment on the price.)

By the way, SPSS coined the phrase “predictive analytics”, with the rest of the industry then coming around to use it. As with all successful marketing phrases, it’s somewhat misleading, in that it’s not wholly focused on prediction.

2) how does it position IBM vs. competitors?

IBM’s ownership immediately makes SPSS a stronger competitor to SAS. Any advantage to the rest of IBM depends on the integration roadmap and execution.

3) How does this particularly affect SAP and SAS and Oracle, IBM’s closest competitors by revenue according to IDC’s figures?

If one of Oracle or SAP had bought SPSS, it would have given them a competitive advantage against the other, in the integration of predictive analytics with packaged operational apps. That’s a missed opportunity for each.

One notable point is that SPSS is more SQL-oriented than SAS. Thus, SPSS has gotten performance benefits from Oracle’s in-database data mining technology that SAS apparently hasn’t.

IBM’s done a good job of keeping its acquired products working well with Oracle and other competitive DBMS in the past, and SPSS will surely be no exception.

Obviously, if IBM does a good job of Cognos/SPSS integration, that’s bad for competitors, starting with Oracle and SAP/Business Objects. So far business intelligence/predictive analytics integration has been pretty minor, because nobody’s figured out how to do it right, but some day that will change. Hmm — I feel another “Future of … ” post coming on.

4) Do you predict further M&A?

Always.

Related links

• Official word from SPSS and IBM
• Blog posts from Larry Dignan and James Taylor
• James Kobelius’s post, which includes the obvious point that Oracle — unlike SAP — has pretty decent data mining of its own
• Eric Lai’s actual article

So anyway, what do you guys have to add to the following observations?

• Super-low-latency financial services industry tasks are often “embarrassingly parallel.” Thus, near-linear scale-out is common.
• That said, good parallelism seems fairly new in CEP engines (of course, CEP engines are fairly new themselves — for all I know, some have been parallel since inception).
• I’ve heard claims of up to 400,000 messages/second/core for simple queries or patterns.
• I’ve heard claims of 70,000 messages/core for not-so-simple queries or patterns, and probably higher than that depending on what the meaning of “simple” is.
• IBM just disclosed >15,000 messages/core on a pretty low-powered processor.
• I’ve heard that Coral8, Apama, and StreamBase rarely lost deals due to performance or throughput problems. I’ve heard that the same is not as true of Aleri.
• StreamBase proudly says it’s been fully multithreaded since academic research-project days.  For Apama multithreading is evidently a more recent feature. But does it matter much?

1.  Just how many processors or cores does it take to get those 5 million messages/sec through? A little birdie says 4,000 cores.

The TD Securities First of a Kind (FOAK) project ran on 356 nodes of a Blue Gene/P supercomputer. The Blue Gene/P may have one or more racks. Each rack consists of 1,024 nodes, each with four Power4 processor cores. The Power4 core was selected as the optimal choice for power consumption and performance, rather than the latest generation of Power chips (Power6). So, the 5 million messages per second were handled on 1,424 Power4 processor cores. This extra information is important so that customers understand the environment, and don’t jump to the conclusion that the latest generation of processor cores (e.g. Power6, or Intel Nehalem, for example) are required to gain this type of performance.

2.  Did the NSA really pay you for Streams, or did you throw your own money into the development, say $35 million?

The US Government has been working with IBM Research since 2003 on a radical new approach to data analysis that enables high speed, scalable and complex analytics of heterogeneous data streams in motion. We will not comment on the particular part of the US Government that we have worked with. Yes, the US Government has a paid contract with IBM Research for this work. IBM will not comment on the amount paid to develop Streams, although we have stated that we have had 40 to 70 people on the team over the last 6 years.

3.  Does Streams have much in the way of the decisioning logic that financial services outfits like for applications like order routing, trade matching, algo trading, etc.?

Streams, via the SPADE programming language, has many commands that simplify creating financial applications. For example,

• Filtering (on things like stock tick)
• Split and Join on streams
• Aggregate to perform sliding window operations based on several parameters like time, count, custom window, etc
• Math functions such as Min, Max, Avg, Sum, Any, First, Last, etc

We’ve also been working with customers and prospects to add User Defined Operators to calculate ‘greeks’ such as Volume Weighted Average Price (VWAP), Daily VWAP, Price to Earnings Ratio, Dividend Yield, Year to Date Return, etc.

*Indeed, as I draft this post-Celtics-game, the embargo is already expired.

Marketing aside, IBM System S/InfoSphere Streams is indeed a CEP/stream processing engine + language (with an Eclipse-based development environment). Apparently, IBM’s thinks InfoSphere Streams (if that’s what it winds up being renamed to) is or will be differentiated from other CEP packages in:

• Scale-out. (That’s the one that appears to be real today. In fact, there’s a prototype running on Blue Gene.)
• Support for complex datatypes such as XML, text, voice, video, etc.
• Security and general industrial-strengthness.
  

IBM Streams seems to be a six-year-old project, with about 15 installations, about half of them true sales. The original and main customer appears to be the US government, with allied governments also in the picture. This usage involves a lot of text/voice/video/whatever analysis, but the technology for that is government-developed; IBM’s own complex datatype technology is one of the features left out to achieve the earlier-than-planned product release. Besides governments, it seems there have been actual InfoSphere Streams sales to universities.

Besides those true sales of System S Streams, there are prototypes and — in a new-to-me phrase — “First of a Kind” projects, which I gather are IP-based services engagements. One is with TD Bank Financial — that’s the one on Blue Gene — handling 5 million messages per second, which is an order of magnitude or so higher than any figure I’ve ever heard from StreamBase, Progress Apama, Aleri/Coral8, or Truviso. Latency is in the 150 microsecond range. IBM believes it is indeed close to a couple of true sales to financial services firms, presumably for super-low-latency algorithmic trading.

As is usually the case for CEP vendors, the financial services market seems to be the only one that cares about super-low latency. Everything else IBM talked about seemed to be in the area of data reduction, although IBM likes to think of that as identifying which data matches certain patterns and then transforming it accordingly. Examples mentioned include:

• Intelligence work
• Identifying marine mammals via sonar
• Neonatal ICU monitoring, being prototyped at the Ontario Institute of Technology
• Wafer testing at IBM’s own Fishkill manufacturing plant
• Info security (in collaboration with IBM’s intrusion detection arm, the former ISS, with a hoped-for first customer at REDACTED)

As I’ve previously noted, the independent CEP vendors aren’t all that active in data reduction, Aleri/Coral8 (somewhat) and StreamBase’s intelligence-community efforts excepted. I’m not in a position to discuss other software generalists’ CEP efforts at the moment, but let’s just say I think data reduction is a generally under-served market, with lots of different niches to fill. IBM System S/InfoSphere Streams was well worth bringing to market.

IBM sent a couple of PDFs with more detail on applications and architecture. I can’t find them online, and I think I have permission to post them, so here they are.  (Edit: Jeff Jones subsequently sent over an official IBM white paper link.)

IBM has its own SQL-based stream processing language called SPADE, with an Eclipse-based IDE. SPADE is not StreamSQL-compatible, although some prototype work suggests IBM could support StreamSQL if necessary. The overall IBM InfoSphere Streams product delivery roadmap is, and I quote IBM’s final slide:

Improved tools

• SPADE 2 Language Specification
• Java based custom analytics

Additional adapters

• MQ, Low Latency Messaging, RSS feeds, Cognos, Mashup Center, WebSphere Business Events, XML

Additional Analytics

• Text, Warehouse, video, audio, dense information grinding

Additional platforms

Windows, Unix vendors, Blue Gene, Cell blades, FPGA

In the first (did I mention it was rushed?) release, System S/InfoSphere Streams supports Intel architectures and Red Hat Linux only.

As Stephen O’Grady points out, Sun’s main strength lay in selling to the large enterprise market. Well, that’s Oracle’s overwhelming focus too. As I noted two years ago:

One Oracle response is to provide lots of add-on technologies for high-end customers, on the database and middle tiers alike. In app servers it’s done surprisingly well against BEA. It’s sold a lot of clustering. And it’s bought into and tried to popularize niche technologies like TimesTen and Tangosol’s.

This all makes perfect sense – it’s a great fit for Oracle’s best customers, and a way to get thousands of extra dollars per server from enterprises that may already have bought all-you-can-eat licenses to the Oracle DBMS. And being so sensible, it fits into the Clayton Christensen disruption story in two ways:

1. Oracle may be helpless against mid-tier competition, but it sure has the high-end core of its market locked up.

2. As one type of technology is commoditized, value is created in other parts of the technology stack.

Oracle’s ongoing acquisition spree in system software, application software, and now hardware just supports that story. MySQL, embedded Java, and so on may be welcome to Oracle as yet more opportunities to tap additional markets — but Oracle’s emphasis is and surely will remain on the large enterprise market.

The next notable point may be found in Larry’s key quote:

… our primary reason for designing our own chips is to build computers with the very best performance, reliability and security available in the market. Some system features work much better if they are implemented in silicon rather than software. Once we own Sun, we’ll be able to plan and synchronize new features from silicon to software, just like IBM and the other big system suppliers. We want to work with Fujitsu to design advanced features into the SPARC microprocessor aimed at improving Oracle database performance. In my opinion, this will enable SPARC Solaris open-system mainframes and servers to challenge IBM’s dominance in the data center. Sun was very successful for a very long time selling computer systems based on the SPARC chip and the Solaris operating system. Now, with the added power of integrated Oracle software, we think they can be again.

Let’s examine that. There are almost no major examples of game-changing silicon-to-software enterprise server integrations. In particular, there aren’t at IBM, whose recent mainframe custom-processor offerings seem to do more as gimmicks to reduce software licensing cost than they do as genuine performance accelerators. And while Netezza and Teradata definitely have custom hardware architectures, they use off-the-shelf parts, including in the processor area.

True, there’s probably a cool story along the lines of highly parallelizing database execution on a single processor, ala Kickfire. But that would require a radical redesign of Oracle’s database software, which surely won’t happen, for two reasons:

• Difficulty.
• The need to keep running on non-Oracle hardware.

On the other hand, you don’t need a game-changing advantage to be the preferred hardware vendor; relatively small advantages are fine. While I don’t have stats, I’d guess IBM gets a lot of the hardware revenue underneath DB2. So the latter part of Larry’s quote, in which he emphasizes the benefit of Oracle’s software biz to Sun’s hardware biz — rather than vice-versa — is not at all implausible.

Finally, Larry said regarding Oracle Exadata:

Exadata is built by HP using Intel microprocessors. We have no plans for a SPARC Solaris version of Exadata. We have an excellent relationship with HP that we expect to continue. … The Sun acquisition doesn’t reduce our commitment to Exadata at all.

To interpret that, recall that the “Exadata” name applies just to the storage-tier part of the product, not the database-tier portion. I would be surprised to learn that Sun’s own storage products are based on SPARC rather than Intel processors. So the “have no plans” part of that quote may remain true for a good long time.

What’s more, it’s not obvious that Exadata will be a high-volume product. It’s very important for Oracle to succeed with Exadata, in order to maintain high-end account control. But there may not be a lot of profit opportunity in backstabbing HP on the Exadata hardware side, especially any time soon.

• DB2 is getting Oracle emulation, which I posted about separately.
• IBM says that it had >50 new DB2 data warehouse customers last year. I neglected to ask how many of these had been general-purpose DB2 customers all along.
• By “data warehouse customer” I mean a user for InfoSphere Warehouse, which previously was called DB2’s DPF (Data Partitioning Feature). Apparently, this includes both logical and physical partitioning. E.g., DB2 isn’t shared-nothing without this feature.
• IBM is proud of DB2’s compression, which it claims commonly reaches 70-80%. It calls this “industry-leading” in comparison to Oracle, SQL Server, and other general-purpose relational DBMS.
• DB2 compression’s overall effect on performance stems from a trade-off between I/O (lessened) and CPU burden (increased). For OLTP workloads, this is about a wash. For data warehousing workloads, IBM says 20% performance improvement from compression is average.
• DB2 now has its version of one of my favorite Oracle security features, called Label Based Access Control. A label-control feature can make it much easier to secure data on a row-by-row, value-by-value basis. The obvious big user is national intelligence, followed by financial services. IBM says the health care industry also has interest in LBAC.
• Also in the security area, IBM reworked DB2’s audit feature for 9.5
• I think what I heard in our discussion of DB2 virtualization is:
  • Increasingly, IBM is seeing production use of VMware, rather than just test/development.
  • IBM believes it is a much closer partner to VMware than Oracle or Microsoft is, because it’s not pushing its own competing technology.
  • Generally, virtualization is more important for OLTP workloads than data warehousing ones, because OLTP apps commonly only need part of the resources of a node while data warehousing often wants the whole node.
  • AIX data warehousing is an exception. I think this is because AIX equates to big SMP boxes, and virtualization lets you spread out the data warehousing processing across more nodes, with the usual parallel I/O benefits.
• When IBM talks of new autonomic/self-tuning features in DB2, they’re used mainly for databases under 1 terabyte in size. Indeed, the self-tuning feature set doesn’t work with InfoSphere Warehouse.
• Even with the self-tuning feature it sounds as if you need at least a couple of DBA hours per instance per week, on average.
• DB2 on Linux/Unix/Windows has introduced some enhanced workload management features analogous to those long found in mainframe DB2. For example, resource allocation rules can be scheduled by time. (The point of workload management is to allocate resources such as CPU or I/O among the simultaneous queries or other tasks that contend for them.) Workload management rules can have thresholds for amounts of resources consumed, after which the priority for a task can go up (”Get it over with!”) or down (”Stop hogging my system!”).

Key points include:

• This really is more like Oracle emulation than it is transparency, a term I carelessly used before.
• IBM’s Oracle emulation effort is focused on two technological goals:
  • Making it easy for an Oracle application to be ported to DB2.
  • Making it easy for an Oracle developer to develop for DB2.
• The initial target market for DB2’s Oracle emulation is ISVs (Independent Software Vendors) much more than it is enterprises. IBM suggested there were a couple hundred early adopters, and those are primarily in the ISV area.

Because of Oracle’s market share, many ISVs focus on Oracle as the underlying database management system for their applications, whether or not they actually resell it along with their own software. IBM proposed three reasons why such ISVs might want to support DB2:

• Oracle is expensive. In particular, IBM suggested it is more flexible on licensing terms for resale than Oracle is. I find that easy to believe.
• Hey, there’s a DB2 market or installed base out there of some size — why not address it?
• Acquisition-fueled expansion in applications makes Oracle a much bigger competitor to many ISVs (all around the world) than it used to be before. That one makes all kinds of sense.

And by the way — if I wanted an Oracle-emulating DBMS, I’d feel a lot happier about doing business with IBM than I would with EnterpriseDB.

IBM feels that DB2’s Oracle compatibility is a strict superset of EnterpriseDB’s, which it presumably has carried over more or less in its entirety. I didn’t press too hard for examples of what Oracle emulation DB2 offers and EnterpriseDB doesn’t, but IBM did say something about support for more programming languages. IBM was clear on one broad area where DB2 does not offer Oracle emulation, which is the specifics of various kinds of datatype support or other specialized data access methods. For example, IBM has its own syntax for querying text, geospatial, or XML data, and has not added support for Oracle’s alternative approaches.