• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

*Links to Gartner Magic Quadrants commonly break, but that one worked at the time of this posting.

• Gartner thinks that data warehouse appliances are on the rise, due to their simplicity.
• Gartner correctly says that Teradata has been a data warehouse appliance vendor from the getgo.
• Gartner characterizes IBM as being an appliance vendor as well.
• Gartner suggests that HP is having trouble living up to its technical promises for Neoview.
• Gartner further suggests — no surprise here — that HP Neoview has had very few new customers past its initial wave.
• Gartner notes IBM’s difficulties in selling data warehouse installations of DB2, despite what on paper is great-sounding technology.
• Gartner says — also no surprise — that illuminate “has seen little success in North America since opening its first office in the U.S. over two years ago.”
• Ingres has evidently gotten a few BI-centric “appliance” deals, e.g. with Jaspersoft. But basically Ingres isn’t doing well in data warehousing.
• Gartner does say Ingres has “the strongest open-source DBMS offering for data warehousing.” Being very literal about “open source,” that’s a defensible claim — but it’s pretty irrelevant in a world where Greenplum Single-Node Edition can be had for free. It also waves away all the data mart use cases in which Infobright Community Edition shines.
• Gartner says that Netezza is working out as a “complex workload” enterprise data warehouse provider, according to reference checks, in addition to its established success in data mart scenarios.
• Gartner says Oracle’s offering has finally become “accepted” in the market for databases >50 TB. I guess I can live with that fairly weak claim, but I wouldn’t go much further than that.
• Gartner asserts that, unlike software-only Oracle, Oracle Exadata isn’t significantly harder to administer than “other mixed OLTP/OLAP DBMS vendors,” because Exadata is fast enough you don’t need to jump through all those hoops any more to get tolerable performance. The money quote is “one reference reported reducing the number of indexes by a factor of 100 to fewer than five.” Note, however, that Gartner does not seem to assert that Exadata’s ease of use rivals that of the newer analytic DBMS specialists.
• Gartner confirms Oracle’s reluctance to do onsite Exadata POCs, but says it is not absolute. This is roughly compatible with what I’m hearing elsewhere, and indeed with Oracle own claims to be ramping up availability of Exadata POC hardware.
• Gartner’s criteria for inclusion include at least 10 different organizations having a product “in production.” Thus, the big surprise was ParAccel being included. The money quote there is “With approximately 20 customers in the pharmaceutical, retail, financial and media/advertising analytics sectors, ParAccel has a good reference base.” That assessment is difficult to reconcile with other information, but I’ve been told Gartner is sticking to its guns. That assessment would be even harder to believe if those 20 references were all alleged to be true production customers.
• Gartner notes that you basically can’t run a 1 TB+ MySQL data warehouse without sharding. (Of course, Infobright has an alternative, and up to a small number of terabytes so does Kickfire.)
• Gartner reports that at least some customers are pleased with Sybase IQ’s mixed workload/enterprise data warehouse capabilities.
• Gartner correctly notes that Oracle Exadata is a price-competition challenge for Teradata.
• Gartner notes that 20% of Vertica’s customers are outside the US. While not shocking, that’s more than I realized.
• Gartner notes something I don’t think I’ve posted yet, which is that Vertica has a customer with 300 TB of data. (The identity is a deep dark secret, but if I told you you probably wouldn’t recognize the name anyway.)

As does any such piece, the Gartner Data Warehouse DBMS Magic Quadrant also has outright errors.  For example:

• Aster Data isn’t really “the newest entrant to the DBMS data warehouse world.”
• Aster’s SQL/MapReduce was not new in Release 4.0.
• Greenplum isn’t yet pushing down code to the storage tier.
• I’m not sure what kind of database-tier parallelism Gartner is claiming is new in Oracle in 11g Release 2 — but I doubt it’s really new. Rather, what Oracle has done recently is make parallelism less administratively cumbersome.
• Vertica wasn’t really the first DBMS in the cloud. At most it was the first pure-play analytic DBMS to get there.

• VectorWise, the product, will be an open-source columnar analytic DBMS. (But that’s not quite true. Pending productization, it’s more accurate to call the VectorWise technology a row/column hybrid.)
• VectorWise is due to be introduced in 2010. (Peter Boncz said that to me more clearly than I’ve seen in other coverage.)
• VectorWise and Ingres have a deal in which Ingres will at least be the exclusive seller of the VectorWise technology, and hopefully will buy the whole company.
• Notwithstanding that it was once named something like “MonetDB,” VectorWise actually is not the same thing as MonetDB, another open source columnar analytic DBMS from the same research group.
• The MonetDB and VectorWise research groups consist in large part of academics in Holland, specifically at CWI (Centrum voor Wiskunde en Informatica). But Ingres has a research group working on the project too. (Right now there are about seven “highly experienced” people each on the VectorWise and Ingres sides, although at least the VectorWise folks aren’t all full-time. More are being added.)
• Ingres and VectorWise haven’t agreed exactly how VectorWise and Ingres Classic will play together in the Ingres product line. (All of the obvious possibilities are still on the table.)
• VectorWise is shared-everything, just as Ingres is. But plans — still tentative — are afoot to integrate VectorWise with MapReduce in Daniel Abadi’s HadoopDB project.

The MonetDB project is led by Martin Kersten, with whom I chatted at SIGMOD in June (standing up and not taking notes, so I may have some details wrong). I get the impression, based on that conversation, my VectorWise call, and other data:

• Martin has been researching analytic DBMS (mainly but not only relational) since the late 1970s, and has been based at CWI since 1985.
• Peter Boncz has been either second in command of that crew or close to it.
• Martin Kersten, Peter Boncz, and the CWI/MonetDB team in general have gotten all sorts of computer science glory for their work.
• Martin has enjoyed generously stable government research funding for his group, but has found commercialization of the technology more difficult than he might at, stay, Stanford. The figure of 15 MonetDB researchers comes to mind, although I see from Martin’s bio that he oversees a team of ~55 in total.
• One early attempt at commercializing MonetDB turned into a company called Data Distilleries that was sold to SPSS. Peter Boncz was chief architect of Data Distilleries.
• Besides VectorWise, there are at least two other recent spin-off companies from the MonetDB project. One is a zero-headcount shell, set up to facilitate MonetDB project members (and others) consulting to users of the open source MonetDB technology. The other is in stealth mode, focusing on some vertical market.

I further get the impression that VectorWise was actually Marcin Zukowksi’s Master’s Ph.D project, with Peter Boncz being his advisor. VectorWise also boasts another Peter Boncz student, who wrote about updating column stores.

As one might expect from the name, VectorWise does vector processing. I.e., the hard part of Marcin’s work was developing vectorized algorithms for one SQL operation after another. Vectorization, pipelining, and FPGAs might all seem to go together — XtremeData certainly seems to think so — but the VectorWise folks preferred to develop for Intel CPUs anyway, for pretty much the usual reasons. Another major theme is trying to get the right things into CPU cache, because in their opinion RAM cache is just sooooo painfully slow.

Our discussion of VectorWise’s compression was interesting. Highlights included:

• The design requirement is that decompression work at a rate of 3 gigabytes/second or so. That way the system is faster overall than if it operated at 1 gigabyte/second on uncompressed data, which I gather is the alternative.
• VectorWise takes 4-5 steps CPU cycles to decompress a tuple.
• VectorWise says it sacrificed compression ratio to achieve speed. That said, VectorWise claims 3-4X compression on TPC-H data, which is no worse than what ParAccel reported, and enjoys higher compression rates on other kinds of data.
• VectorWise decompresses data before manipulating it, and claims that the advantages of operating on compressed data are only significant if — like Vertica but apparently unlike VectorWise — the database stores columns in multiple sort orders each.
• VectorWise’s compression is mainly on numerical and numerical-like (e.g. date) datatypes. An exception is that VectorWise uses dictionary compression on string data when it makes sense to do so.

Other notes include:

• VectorWise has technology akin to Microsoft SQL Server’s Shared Scans, in which multiple queries that require similar table scans don’t have to repeat all the redundant scanning work. I need to get better at figuring out which other analytic DBMS do similar things.
• While VectorWise hasn’t yet been open-sourced, its code is in the hands of some other academic institutions, used mainly for computer science research (as opposed to, say, as a data store for some kind of scientific experiment).
• VectorWise’s scalability has only been tested up to eight cores.

• Ingres is much of the way toward SQL 92 compatibility.
• Ingres also has added some industry-common proprietary SQL extensions. E.g., it appears multiple vendors have adopted Oracle’s way of doing substrings along with the standard way.
• Development chief Bill Maimone points out that when Ingres was still owned by CA, a lot of work was done to increase availability. (Arguably, Ingres’ main use at CA was as the underpinning for CA Unicenter.)
• I’m sure Ingres would disagree with this phrasing, but it sounds as if Ingres’ appliance strategy is more bait-and-switch than anything else. E.g., Ingres appliances don’t have the Red Hat version of Linux many users want.
• That said, Ingres’ BI appliance has enjoyed a “fair” number of small sales. Ingres claims – not unreasonably — that with a mature optimizer and parallel query, its product is suitable for at least lightweight reporting despite a lack of more advanced data warehousing features.
• Ingres does say, using the appliances as an example, that it built an installer much easier than what it had before.
• Although Ingres said something last August to Seth Grimes that sounded like it was going to integrate MonetDB, that’s not actually happening. Ingres does seem to have some kind of data warehousing speed-up project in the works, but was coy as to details.
• Ingres believes it will soon leapfrog closed-source vendors in its geospatial support, helped by significant community contributions. (Geospatial was the only part of Ingres it didn’t have the licensing rights to open-source, which is a bit part of the reason it’s being replaced.)
• However, it didn’t sound as if Ingres was doing much else in datatype extension.
• Speaking of data-warehousing-oriented enhancements, another example of nontrivial community contribution is partition-compatible hash aggregates.
• Another example is column encryption.
• Generally, Ingres echoes what other open source vendors say: The community will work on what it wants to work on, and many of the vendor’s priorities have to be implemented by the vendor itself.
• I didn’t ask directly, but I get the sense Ingres has concentrated some of it sales efforts in the financial industry, for three reasons. First, Ingres says it is doing a lot of Sybase replacements. Second, when Ingres said it hasn’t had as much success replacing Oracle, but has some hopes that this will change, it used financial companies as an example. Third, that’s the market Ingres used to say it was focusing on.
• Ingres says it also is doing OK in Microsoft replacements. But MySQL replacements are harder because MySQL is so non-standard that porting applications is more challenging.
• In another slap at MySQL, Ingres suggested that it’s hard to reference a BLOB in MySQL without pulling the whole thing down to the client.
• While conceding Postgres is good technology, Ingres asserted having a support organization is a competitive advantage. I asked a couple of times “What about EnterpriseDB?”, but didn’t get a response.

]]> http://www.dbms2.com/2009/04/02/ingres-update/feed/ 6 http://www.dbms2.com/2009/03/18/database-implications-if-ibm-acquires-sun/ http://www.dbms2.com/2009/03/18/database-implications-if-ibm-acquires-sun/#comments Wed, 18 Mar 2009 14:48:13 +0000 Curt Monash http://www.dbms2.com/?p=722 Reported or rumored merger discussions between IBM and Sun are generating huge amounts of discussion today (some links below). Here are some quick thoughts around the subject of how the IBM/Sun deal — if it happens — might affect the database management system industry.

• IBM is already serious about supporting multiple database management systems. DB2 on open systems is IBM’s flagship DBMS. But DB2 on mainframes and at least one flavor of Informix seem to be getting maintained and enhanced fairly seriously as well. And IBM has further DBMS products as well (e.g., DB/2 on the AS/400). There’s little reason to think IBM would orphan MySQL or any other DBMS product.
• IBM is very open-source-friendly. For a company that grew up for decades on proprietary software — and still is a huge software products vendor — IBM is very serious about open source. If you doubt that, I have two words for you: “Linux” and “Eclipse”.
• MySQL might finally get its industrial-strength act together. IBM is good at database management and good at open source. MySQL becoming a no-apologies transactional DBMS would obviously put pressure on Ingres, PostgreSQL, and EnterpriseDB, although there surely would be lots of happy talk about the open source DBMS market being validated, lifting all the vendors and so on. Also, a better MySQL could be bad news for Microsoft SQL Server too.
• Sun has a lot of DBMS partnerships right now. Obviously, Sun owns MySQL, and has partnerships with MySQL storage engine vendors such as Infobright and Kickfire. Sun also has a substantial partnership with Greenplum, and a Barneyesque* one with ParAccel. And of course Sun has strong working relationships with major database vendors such as Oracle and Sybase. What’s more, on a case-by-case basis, Sun may cooperate in the field with yet other DBMS sellers. E.g., I’ve confirmed at least one instance of a Sun sales rep recommending a Kognitio DBMS.
• IBM partners with outside DBMS vendors too. You’d think IBM’s gazillion DBMS product lines would be enough. But nooooo. I frequently hear rumblings of IBM’s hardware or services operations working with other DBMS products as well. (This is, of course, actually to their credit.)
• Short-term, there probably would be little effect on partnerships. Greenplum runs on Sun’s Thumper/Thor line of boxes. DB2 doesn’t, and certainly isn’t optimized for same. In the short term, to sell Thors, Sun would presumably continue to sell Greenplum.
• Longer-term, there could be a DBMS rationalization. DB2, Informix, MySQL + storage engines, and big independent vendors such as Oracle and Sybase would surely always get attention. That’s a lot. There might not be room for much mind share for many database products and vendors beyond that list.

*A Barney partnership is one in which two or more vendors get on stage and do a song and dance about how much they love each other, with little substance beyond that.

Related links

• Larry Dignan thinks the IBM/Sun deal is sensible and ripe to happen.
• Dana Gardner thinks otherwise.
• Matt Asay seems to agree that IBM understands the open source business.
• Before IBM acquired it, solidDB was scheduled to provide a serious MySQL transaction processing engine.

Point 23 says “The average tenure of an Ingres Engineer is 15+ years.”  On the other hand, Point 3 says “The longest serving member of Ingres staff is John Smedley who has been with us since June of 1987.”  And most of Ingres’ technical staff left after Ingres was acquired by CA, which occurred a few months shy of 15 years ago. Reconciling all that is challenging.

Actually, I was dubious about a second claim too, namely that Ingres/Star was the first distributed DBMS; I thought that the distributed version of Tandem NonStop SQL actually predated it by a few years. But a somewhat contemporaneous article with a number of distributed DBMS dates shows my memory was wrong on that score.

Highlights of this year’s data warehouse DBMS Magic Quadrant include:

• Teradata is #1, Oracle is #2, and IBM is #3, with the first two if anything slightly extending their leads.  (in 2006, IBM was #2.)
• Netezza has been given a nice upwards (actually, more rightwards) bump and is now a clear #4.
• Microsoft is treading water at a clear #5.
• Greenplum and Sybase have slid back some, but depending on which dimension you weight more heavily are somewhere in the #6-8 range.
• HP joins newly, as the other #6-8 competitor, a little behind Sybase.
• Vertica joins as a first-timer, as a clear #9.
• Kognitio and SAND are next, with hefty gains in “ability to execute”, both leapfrogging Sun/MySQL.
• Ingres, iLLuminate, and 1010data straggle in at the bottom, all of them new (at least versus 2006-7).

I don’t really have a lot of quarrel with the “completeness of vision” rankings.  As I see it, important attributes of a data warehouse DBMS “vision” would include:

• A performance story across at least a reasonable range of workloads.
• Either a clear hardware architecture story, or else a clear story as to why hardware architecture is relatively unimportant.
• SQL 2003 and further features in integrated analytics.
• Reasonable OLTP-like features, from the basics — ACID compliance! — to manageability, high availability and fast-enough update/load.
• Good compatibility with third-party products.

Gartner’s rankings are not ridiculous by those standards.  Aster would surely have ranked high, but obviously they did not meet the confirmed-sale requirements for inclusion.

So what about Gartner’s “ability to execute” rankings?  These are approximately:

• Teradata at #1
• Oracle and IBM tied at #2-3
• HP, Sybase, Microsoft, and Netezza tied at #4-7
• Greenplum at #8, Vertica at #9, and everybody else trailing after

That looks like it’s basically a measure of revenue, blending overall corporate and data-warehouse-DBMS-specific figures in some way, adjusted for who can deploy the most credible-sounding executive who appears to simultaneously have his — I use the male pronoun deliberately — finger on development and revenue-generation alike.

Frankly, I think it’s that dimension that makes Gartner Magic Quadrants well-nigh meaningless.  If you asked me in which vendor’s execution-on-vision I had the most confidence, I’d stammer around unless I felt free to reframe the question and shoot back “Which PART of the vision?”  If you want to deploy a 1 terabyte data warehouse with a highly diverse workload — well, Oracle, IBM, Teradata, and to a lesser extent Microsoft have been doing that for years, and they deserve to be atop the ability-to-execute charts, with Netezza perhaps not far behind.  If you want to run fast queries on cheap hardware on 200 GB of data, Sybase IQ is a proven market leader.  If you want a cheap 100 TB data warehouse that will soon scale to over a petabyte, Oracle’s great achievements in other areas of DBMS and its clever Exadata ideas suffice merely to put it on a par with those smaller vendors that have actually deployed a few such systems each, albeit behind Teradata.

When selecting a database management system for analytic processing, confine yourself to those vendors whose products can, today, do everything you’re likely to need for the next few years. Further require that they be on track to soon deliver most of what you seriously want over that time period.  And throw the Gartner MQ into the nearest bit bucket, before it confuses your evaluation cycle irredeemably.

But for most applications at any enterprise – and for all applications at most enterprises – super high-end DBMS aren’t required. There are relatively few applications that wouldn’t run perfectly well on PostgreSQL or EnterpriseDB today. Ingres and Progress OpenEdge aren’t far behind (they’re a little lacking in datatype support). Ditto Intersystems Cache’, although the nonrelational architecture will be off-putting to many. And to varying degrees, you can also do fine with MySQL, Pervasive PSQL, MaxDB, or a variety of other products – or for that matter with the cheap or free crippled versions of Oracle, SQL Server, DB2, and Informix.

What’s more, these mid-range database management systems can have significant advantages over their high-end brethren. The biggest is often price, for licenses and maintenance alike. Beyond that, they can be much easier to administer then their more complex counterparts. For example, Progress OpenEdge and Informix SE have long been reseller favorites, in large part because they can be installed at small businesses and locations that lack technical staff, and rarely if ever require DBA attention. Programming and hardware costs can sometimes be lower as well.

And what these mid-range DBMS don’t do today, they likely will do soon. In the 1990s, Microsoft SQL Server was the mid-range entry threatening to disrupt the market. But it’s grown up quite nicely. EnterpriseDB is equal or superior in every way I can think of to Oracle7, a few security certifications perhaps excepted. (They’d probably argue the release number in that claim should be 1 or 2 higher, but I’d have to compare their multimedia support to what I recall of Oracle 8.1.5 before I agreed.)

Will these mid-range database management systems truly “disrupt” the DBMS market, as many open source advocates hope? Or will they be largely co-opted into the oligopoly, as Microsoft SQL Server was? That’s a discussion for another time. For now, please just keep your mind open to DBMS alternatives – the high-end approach is not always the best.

EDIT: For a contrary view, please see my follow-up post making the opposite case.

Please subscribe to our feed!

I find myself defining a new product category – midrange OLTP/multipurpose DBMS. (Or just midrange DBMS for brevity.) Nothing earthshaking here; I’m simply referring to those products that:

• Match the OLTP DBMS state of the art of some point in the 1990s, and hence meet most application needs today.
• Are significantly more affordable (in license, maintenance, and/or administrative costs) than the top-end products from Oracle, Microsoft, and IBM.

Basically, these come in three groups.

1. “Open source” – these are the first ones most people think of. MySQL leads the way – if 5.0 works as advertised. EnterpriseDB has a very interesting offering. PostgreSQL has some fans away from EnterpriseDB. And then there are Ingres, Firebird, and others, at various levels of open-source purity.
2. “Standard editions” – Informix SE may have been the first DBMS to use the “Standard Edition” term, but Oracle, IBM, and Microsoft all have free or low-cost versions of their flagship DBMS. These are basically crippled versions of the lead products, conceived as entry-level products to fend off price competition.
3. VAR-oriented – Progress OpenEdge and InterSystems Cache’ are the two biggies in this category. (Informix SE also used to be pretty big.) These are products that are great to leave running with a stable packaged application suite in a small enterprise, or with a single custom app in a large department. I’m visiting Progress on Monday and may have more to say after that.

Even that’s a far from complete list. For example, the Sybase iAnywhere core product used to be a great laptop/desktop/workgroup RDBMS, and Sybase makes periodic noises about recommitting to that market.

So should you care? Yes. Overall, there’s a swelling, rather classical, disruption story. And if you’re an enterprise looking to consolidate your DBMS suppliers, you would do well to use one of these midrange products as well as (if you even need it) one of the top-end OLTP RDBMS.

As for what these midrange DBMS are or can be used for – well, that’s a long list. But some of the broad categories are:

• Most web-facing apps.
• Most departmental apps.
• All the processing at a typical medium or small enterprise.

Of course, the details depend on the match between your application needs and particular products. As just one example — if you need non-tabular datatype support, such as XML or geospatial, over half the possibilities can be immediately thrown out.

• Confirms that EnterpriseDB is OLTP-focused, unlike Greenplum. That said, they are also used for some reporting and so on. But they don’t run 10s-of-terabytes sized data marts.
• Claims EnterpriseDB has a high level of Oracle compatibility – SQL, datatypes, stored procedures (so that would be PL/SQL too), packages, functions, etc.
• Claims ANTs isn’t nearly as Oracle-compatible.
• Claims 50-100% better OLTP performance out of the box than vanilla PostgreSQL, due to auto-tuning.

Also, EnterpriseDB has added a bunch of tools to PostgreSQL – debugging, DBA, etc. And it provides actual-company customer support, something that seems desirable when using a DBMS. It should also be noted that the product is definitely closed-source, notwithstanding EnterpriseDB’s open-source-like business model and its close ties to the open source community.

Basic stats Andy offers include: EnterpriseDB just started selling in January, 2006. They now have 75 customers, single millions of dollars in sales, 140 people worldwide in six countries.

A lot of these customers are ISVs, looking for a cheaper alternative to bundling Oracle. This echoes Ingres’ story; such new-name business as Ingres has seems to come mainly from ISVs these days. And a few customers are just pure PostgreSQL users, who don’t care about Oracle compatibility but want support.

As for the enterprise customers, Andy cites a variety of replacements of other DBMS, but concedes that many sales are for new and/or proof-of-concept applications. It should also be noted that enterprise customers typically pay on a subscription basis, which obviously depresses current revenue (but does make EnterpriseDB pretty comparable to the substantially lager MySQL, apples-to-apples).