• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

*Links to Gartner Magic Quadrants commonly break, but that one worked at the time of this posting.

• Gartner thinks that data warehouse appliances are on the rise, due to their simplicity.
• Gartner correctly says that Teradata has been a data warehouse appliance vendor from the getgo.
• Gartner characterizes IBM as being an appliance vendor as well.
• Gartner suggests that HP is having trouble living up to its technical promises for Neoview.
• Gartner further suggests — no surprise here — that HP Neoview has had very few new customers past its initial wave.
• Gartner notes IBM’s difficulties in selling data warehouse installations of DB2, despite what on paper is great-sounding technology.
• Gartner says — also no surprise — that illuminate “has seen little success in North America since opening its first office in the U.S. over two years ago.”
• Ingres has evidently gotten a few BI-centric “appliance” deals, e.g. with Jaspersoft. But basically Ingres isn’t doing well in data warehousing.
• Gartner does say Ingres has “the strongest open-source DBMS offering for data warehousing.” Being very literal about “open source,” that’s a defensible claim — but it’s pretty irrelevant in a world where Greenplum Single-Node Edition can be had for free. It also waves away all the data mart use cases in which Infobright Community Edition shines.
• Gartner says that Netezza is working out as a “complex workload” enterprise data warehouse provider, according to reference checks, in addition to its established success in data mart scenarios.
• Gartner says Oracle’s offering has finally become “accepted” in the market for databases >50 TB. I guess I can live with that fairly weak claim, but I wouldn’t go much further than that.
• Gartner asserts that, unlike software-only Oracle, Oracle Exadata isn’t significantly harder to administer than “other mixed OLTP/OLAP DBMS vendors,” because Exadata is fast enough you don’t need to jump through all those hoops any more to get tolerable performance. The money quote is “one reference reported reducing the number of indexes by a factor of 100 to fewer than five.” Note, however, that Gartner does not seem to assert that Exadata’s ease of use rivals that of the newer analytic DBMS specialists.
• Gartner confirms Oracle’s reluctance to do onsite Exadata POCs, but says it is not absolute. This is roughly compatible with what I’m hearing elsewhere, and indeed with Oracle own claims to be ramping up availability of Exadata POC hardware.
• Gartner’s criteria for inclusion include at least 10 different organizations having a product “in production.” Thus, the big surprise was ParAccel being included. The money quote there is “With approximately 20 customers in the pharmaceutical, retail, financial and media/advertising analytics sectors, ParAccel has a good reference base.” That assessment is difficult to reconcile with other information, but I’ve been told Gartner is sticking to its guns. That assessment would be even harder to believe if those 20 references were all alleged to be true production customers.
• Gartner notes that you basically can’t run a 1 TB+ MySQL data warehouse without sharding. (Of course, Infobright has an alternative, and up to a small number of terabytes so does Kickfire.)
• Gartner reports that at least some customers are pleased with Sybase IQ’s mixed workload/enterprise data warehouse capabilities.
• Gartner correctly notes that Oracle Exadata is a price-competition challenge for Teradata.
• Gartner notes that 20% of Vertica’s customers are outside the US. While not shocking, that’s more than I realized.
• Gartner notes something I don’t think I’ve posted yet, which is that Vertica has a customer with 300 TB of data. (The identity is a deep dark secret, but if I told you you probably wouldn’t recognize the name anyway.)

As does any such piece, the Gartner Data Warehouse DBMS Magic Quadrant also has outright errors.  For example:

• Aster Data isn’t really “the newest entrant to the DBMS data warehouse world.”
• Aster’s SQL/MapReduce was not new in Release 4.0.
• Greenplum isn’t yet pushing down code to the storage tier.
• I’m not sure what kind of database-tier parallelism Gartner is claiming is new in Oracle in 11g Release 2 — but I doubt it’s really new. Rather, what Oracle has done recently is make parallelism less administratively cumbersome.
• Vertica wasn’t really the first DBMS in the cloud. At most it was the first pure-play analytic DBMS to get there.

Greenplum Single-Node Edition:

• Is free of charge, although you can buy support.
• Has no restrictions on use, production or otherwise.
• Has no restrictions on database size.
• Is closed-source.

For those who want free, terabyte-scale data warehousing software, Greenplum Single-Node Edition may be quite appealing, considering that the main available alternatives are:

• General-purpose open-source DBMS, such as PostgreSQL and MySQL (lacking analytic DBMS performance and features)
• Infobright Community Edition (the other best choice – Infobright’s commercial sales success indicates the solidity of Infobright’s technology)
• Rough research-project code and other other questionable open source offerings
• Crippleware from other commercial analytic DBMS vendors (e.g., Teradata)

For example, comparing PostgreSQL-based Greenplum with PostgreSQL itself, Greenplum offers:

• The ability to scale out queries across all cores in your box (and no, pgpool is not a serious alternative)
• Storage alternatives such as columnar (I am told that EnterpriseDB recently stopped funding a project for a PostgreSQL columnar option)

Greenplum would surely also argue that its software is superior to PostgreSQL in parallel load, compression, MapReduce integration, and general fit-and-finish. I imagine that in some (perhaps not all) cases it would be right. PostgreSQL’s main technical advantages over Greenplum would probably lie in the area of datatype extensibility.

The main target users for Greenplum’s Single-Node Edition are obviously individual enterprise power users or very small analytic teams. I.e., it’s people with a data mart need that a central data warehouse isn’t meeting. Potential benefits to Greenplum include:

• Adding value to its Enterprise Data Cloud story
• Seeding the market for future enterprise sales
• Depriving competitors of revenue, perhaps at enterprises too small to ever be paying Greenplum customers

In addition, I see free Greenplum as a charity offering that could be appealing to scientists who face PostgreSQL performance limitations.

Related links

• Greenplum Free Single-Node Edition press release (I’m quoted)
• MySQL Performance blog on MonetDB and Infobright community edition
• PostgreSQL’s restriction to one core per query
• Infobright’s restriction to one core per query

• Greenplum is getting 12-15 new (paying) customers per quarter, all of whom it fondly refers to as “Tier 1″ enterprises.
• Greenplum will hit the 100+ customer mark this quarter (thus joining Vertica and Infobright).
• <10% of Greenplum business is now “influenced” by Sun hardware.

I asked Ben Werther to unpack that last claim for me. He quickly noted that it wasn’t his slide, but rather had been put together by colleagues. That said:

• As of the past quarter or two, <10% of Greenplum’s sales activity is on Sun, which works out to maybe one sale per quarter and at most a small number of sales cycles. (That’s down from from 50%+ not that long ago.)
• Most Greenplum business is now on HP or Dell equipment.  Some is on IBM. There are some interesting sales cycles on Cisco’s new UCS (Unified Computing System) blades, but no closed deals yet. EMC seems to be part of the Cisco story.

No doubt part of the reason for the move away from Sun equipment is the impending Oracle acquisition. Another may be that the Greenplum/Sun appliance is somewhat underpowered. E.g., without particularly high levels of compression, eBay puts over 60 terabytes of data on each Greenplum node, which probably isn’t ideal from the standpoint of query performance.

Greenplum also says that 50% or so of sales are subscription-priced, rather than perpetual-licensed. I don’t have a sense for how long that’s been going on. (Edit: Ben Werther tells me this has been true for over a year.)

• MapReduce is something very new
• MapReduce involves strict adherence to the Map-Reduce programming paradigm
• MapReduce is a single technology

So let’s give it a try.

When Dave DeWitt and Mike Stonebraker leveled their famous blast at MapReduce, many people thought they overstated their case. But one part of their story – one that both Mike and Dave say was most central to their case – was never effectively refuted, namely the claim that these ideas aren’t particularly new. I haven’t actually read enough computer science literature to have an independent opinion on that issue. But I’ll say this – claims from companies such as SenSage, Oracle, or Splunk that “We’ve been doing MapReduce all along” seem pretty credible to me.

True, what those companies were doing things may not have looked exactly like the instant-classic MapReduce programming paradigm. But the same is true of many things almost everybody would agree count as MapReduce. In particular, it is often not the case that you alternate Map and Reduce steps, each of whose outputs is a set of simple <Key, Value> pairs, with data redistributed based on Key at every step.

Here are some examples of what I mean, drawn from my recent MapReduce webinar.

• If you do text indexing in MapReduce, your goal is to wind up with a text index. So at some point you Reduce to a pair <WordName, {all the (DocumentID, offset) pairs for the whole corpus, suitably ordered}>. That’s a heckuva compound “Value”.
• The goal of data mining is usually to estimate a rather small number of parameters based on a large overall data set, often – depending on algorithm – in the form of a single vector. When you do that in MapReduce. you partition data among nodes, calculate something on each node that is structured more or less like your final vector. So when it comes time for the reduce, you just ship all of your vectors – one per node – to a single Reduce node, and do the appropriate math. Redistribution based on Key would be quite pointless.
• When you sessionize clickstream logs in MapReduce, you may have just as many output records as input records. However, they now are reformatted, and might have a SessionID appended. In those cases, Reduce isn’t doing much by the way of reduction.
• And as I happens in some Vertica-Hadoop use cases around mortgage trading, sometimes MapReduce can even make data sets vastly larger.

By no means do I think this is a weakness of the MapReduce programming paradigm. Rather, I think it’s a MapReduce strength. But it’s not quite the way MapReduce has been promoted and explained to the IT public.

Finally: MapReduce, as commonly conceived, spans two different – albeit closely related – technology domains:

• Parallel programming
• Distributed data management

For example, I imagine Greenplum’s and Vertica’s MapReduce/SQL combined syntaxes are very similar to each others. But Vertica’s data management implementation of MapReduce, which relies on Hadoop, is very different from Greenplum’s, which is tied into the Greenplum DBMS. Similary, non-DBMS MapReduce implementations are commonly associated with distributed file systems – notably HDFS (Hadoop Distributed File Systems) or Google’s internal GFS (Google File System). In those systems, the parallel language execution part should be aware of how the distributed file management part works – but perhaps that awareness can be pretty lightweight.

Right now, this is a distinction pretty much without a difference. If you choose an implementation of MapReduce — like pure Hadoop (say in the Cloudera distribution) or Hadoop-Vertica or Aster Data’s SQL/MapReduce – you’re basically picking an entire technology stack. But those stacks are going to do a whole lot of changing and maturing in the near future – and as they do, it’s likely that projects will interact or even combine in all sorts of interesting ways.

Bottom line: There are a lot of different ways to exploit MapReduce-related technology.

Highlights include:

• Column orientation is a special case of what Greenplum is calling Polymorphic Data Storage.*
• As per product management chief Ben Werther’s blog post, what Greenplum’s polymorphic data storage boils down to is that you can store different tables in different storage paradigms. This is transparent to the SQL or any other API; it’s just a performance choice.
• Indeed, Greenplum lets you store different partitions of the same table in different storage and/or compression schemes. So Greenplum now has a kind of ILM (Information Lifecycle Management) story, although it doesn’t offer the faster vs. cheaper storage media differentiation options of Sybase IQ or Vertica.
• Greenplum now has, depending on how one counts, three or four main types of table:
  • Traditional PostgreSQL, which has been available since Day One
    
  • Row-oriented append-only (compressible and scan-optimized), available since Greenplum 3.2 (July, 2008)
  • Columnar append-only (new in Greenplum 3.3.4, shipping now)
  • External, in which Greenplum treats something external – in a relational DBMS or otherwise – as if it were a Greenplum table
• Greenplum offers multiple versions of LZ (Lempel-Ziv) and gzip compression, any of which you can choose on a table-by-table or partition-by-partition basis.
• Greenplum offers the same compression algorithms for both row-oriented and column-oriented tables.
• Greenplum says that compression is typically at least 50% better (i.e., to 2/3 as much space) in columnar vs. row storage, for the same algorithm.
• Just as it doesn’t offer columnar-specific compression algorithms, Greenplum also doesn’t sport other columnar features Daniel loves, such as in-memory compression or late materialization. (But then, VectorWise doesn’t do in-memory compression either, and Daniel likes VectorWise.)
• All the Greenplum choices I’ve mentioned have to be made manually by DBAs.
• Similarly, I doubt Greenplum can match Vertica’s engineering for getting updates and trickle feeds quickly into a column store – a traditional columnar Achilles heel that Vertica has invested a lot of effort to circumvent.

*The term “polymorphic” is somewhat, shall we say, overloaded these days.

One of my favorite pages on the Monash Research website is the list of many current and a few notable past customers. (Another favorite page is the one for testimonials.) For a variety of reasons, I won’t undertake to be more precise about my current customer list than that. But I don’t think it would hurt anything to list the analytic/data warehouse DBMS/appliance specialists in the group. They are:

• Aster Data
• Greenplum
• Infobright
• Kickfire
• Kognitio
• Microsoft
• Netezza (my biggest client this year, probably, because of all the Enzee Universe appearances)
• Sybase
• Teradata
• Vertica
• Attivio, which may or may not be construed as being in the analytic DBMS business
• Clearpace, ditto

All of those are Monash Advantage members.

If you care about all this, you may also be interested in the rest of my standards and disclosures.

• Aster announced today that it’s providing .NET support for SQL/MapReduce. Perhaps not coincidentally, Aster’s biggest customer is MySpace, which is apparently a big Microsoft shop.  (And MySpace parent Fox Interactive Media is a SQL/MapReduce fan, albeit running on Greenplum.)
• Aster generally puts more emphasis on MapReduce than SQL/MapReduce rival Greenplum.  That’s a non-trivial comparison, because Greenplum is making progress in SQL/MapReduce itself.
• When talking with Aster folks, I can’t get them to shut up hear a lot about SQL/MapReduce.

I was a big fan of SQL/MapReduce when it was first announced last August. Notwithstanding persuasive examples favoring pure DBMS or pure MapReduce over DBMS/MapReduce integration, I continue to think the SQL/MapReduce idea has great potential.  But I do wish more successful production examples would become visible …

Client experience teaches me that this definition is easy to forget, so let me reemphasize the key point:

Per-terabyte pricing is based on a calculated figure.  Per-terabyte pricing is not based on the current disk space used by your database when managed by the DBMS you are replacing.

There’s at least one important difference in how Vertica and Greenplum calculate database size.  No matter how many times you copy the data, Vertica only charges you for it once.* But if you spin out data marts and recopy data into it — as Greenplum rightly encourages you to do — Greenplum wants to be paid for each copy.  Similarly, Vertica charges only for deployment, and not for test or development; I didn’t remember to ask what Greenplum’s policies are in those regards. (Edit: Greenplum says in a comment below that it doesn’t charge for test or development data either.)

*That policy is a great fit with Vertica’s performance recommendation that you should store columns in different sort orders, perhaps an average of two copies per column.