*I probably have it worse than most people in that regard, because my clients run tentative feature lists and announcement schedules by me well in advance, which may get changed multiple times before the final dates roll around. I also occasionally miss some detail, if it wasn’t in a pre-briefing but gets added at the end.

Anyhow, the three big themes of this month’s announcements are probably:

• Integrating different kinds of analytic processing into databases and DBMS.
• Taking advantage of hardware advances.
• Playing catchup in areas where small vendors’ products weren’t mature yet.

For example, the three biggest data warehouse DBMS product announcements this month are probably:

• Aster Data nCluster 4.5. Much like Aster’s prior release — Aster Data nCluster 4.0 – Aster Data nCluster 4.5 has a major focus on integrating analytics and database processing. This time, the emphasis is on application development tools and pre-built analytic packages. In addition, Aster’s management tool GUIs have been upgraded, building on catch-up functionality in the Aster Data nCluster 4.0.
• Netezza’s “i” add-on to its existing TwinFin products. With Netezza TwinFin(i), Netezza becomes the second MPP RDBMS vendor with a comprehensive “Big Data Analytic Platform” kind of strategy. (Netezza would surely argue that it was the first, but that depends on how seriously one took Netezza’s prior attempt.) Many of the details are different from Aster’s, of course, but the general philosophy is similar. So far, Netezza has announced one interesting proprietary library of analytic packages (for linear/matrix algebra), plus the port of 4,000 or so functions in open source libraries.
• Vertica 4.0. Vertica has had a highly innovative columnar DBMS architecture from the getgo, but at the cost of some restrictions or awkwardness in the relationship between data layout and SQL processing. Vertica says that Vertica 4.0 fixes all that. In addition, it has some analytic processing enhancements, especially in the time series area, where Vertica doesn’t vigorously dispute that Sybase IQ previously had an advantage.

In addition,

• Teradata is announcing its Data Warehouse Appliance 2580, the successor to the Teradata 2550. This is purely a hardware refresh; Teradata’s hardware and software upgrades are not generally synced. The Teradata 2580 upgrades CPUs from Harpertown to Nehalem, includes 3X the RAM of its predecessor, and offers an option for 1 TB disks (thus lowering the bottom price/TB a lot, to $31K list).
• Aster, Vertica, and ParAccel have all called attention to the fact that, if solid-state drives have interfaces like those of disk drives, and if a DBMS supports disk drives, then a DBMS also supports solid-state drives as well. At least Aster and ParAccel have signaled that they have at least one customer or prospect each interested in Fusion I/O’s solid-state technology, especially in the retail sector. This is basically a hardware matter as well, and a big deal only for those who were somehow unaware of the impending dominance of solid-state memory technology.
• Sybase announced its Aleri acquisition earlier this month.
• Various vendors have bragged about various rankings, awards, or benchmarks, or – sometimes less tediously — about last year’s sales results.

For starters, there’s a lot of new analytic functionality. This isn’t Aster/Netezza-style ambitious. Rather, there’s a lot more SQL-99 functionality, plus some time series extensions of the sort that financial services firms – an important market for Vertica – need and love. Vertica did suggest a couple of these time series extensions are innovative, but I haven’t yet gotten detail about those.

Perhaps even more important, Vertica is cleaning up a lot of its previous SQL optimization and execution weirdnesses. In no particular order, I was told:

• Vertica’s delete performance is up “literally” 30-100X, at least in the case of “large” deletes. Performance for “large” updates has been enhanced as well.
• Vertica has finally cleaned up all vestiges of its prior bias to star schemas. For example, Vertica concedes that its product previously would sometimes force a star execution plan that wasn’t really appropriate.
• It is no longer the case that you need to define projections before you load a table into Vertica. This is now fully automatic.
• Vertica 4.0 automatically redesigns the database when new nodes are added to the system.
• When a database designer does hand-tune projections – and there’s no shame in this still being a possibility in Vertica 4.0 – that hand-tuning is now pulled back into the automatic generation/recommendation/whatever wizards for further projections. I.e., there’s a kind of DBA round-trip engineering going on.
• Vertica used to require that tables being joined be identically “segmented” (I think this means distributed across joins). That is no longer the case in 4.0.
• In connection with this new-found flexibility, Vertica now supports full outer joins directly, rather than requiring the left outer join/right outer join/UNION kluge.
• The Vertica 4.0 optimizer is smarter than its predecessor about things like predicate pushdown into subqueries, or exploiting commonality between predicates and partition keys.
• There’s a fundamental change that I don’t understand very well in the Vertica execution engine basic unit of work. It sounds as if in the past all the disk-based data containers the query needed got opened at once and read into memory, whether or not there was enough RAM and CPU cores to handle them, and this problem has now been fixed.
• Vertica always seemed to say that you could query immediately on new data, because even if it hadn’t hit disk yet – the ROS (Read-Optimized Store) – it was available in memory – the WOS (Write-Optimized Store). And queries were in essence federated between the ROS and WOS. But apparently it’s a new feature in Vertica 4.0 that you can read totally fresh data without locking. I confess to not understanding this very well either. (It has something to do with what  Vertica calls “Epochs”.)
• Temporary tables can now be created in Vertica on a local/session basis without any DDL. Make temporary tables easier and more performant is important for a variety of reasons:
  • Microstrategy, Company V* et al. use lots of temp tables. E.g,, Company V on Vertica has 3000 permanent tables and 5-7000 temporary ones.
  • Vertica rightly points out that temporary tables are also important for ELT (Extract/Load/Transform).
  • Vertica further says that single-node OEMs such as security appliance vendors use lots of temp tables.

*Company V = one of the more prominent vertical-market application providers.

In other Vertica highlights:

• It sounds as if 4.0 is the first Vertica release with what I would regard as serious workload management.
• While Vertica has stored and retrieved Unicode since Vertica 3.5 or so, 4.0 will be the first Vertica release in which Unicode is sorted and collated properly.
• Stored-procedure-like functionality is still a future for Vertica.

• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

If a recent Vertica press release is to be believed, Sybase got clobbered. The meat is:

…  Sybase has admitted that under the claim construction order issued by the Court on November 9, 2009, “Vertica does not infringe Claims 1-15 of U.S. Patent No. 5,794,229.” Sybase further acknowledged that because the Court ruled that all the remaining claims in the patent (claims 16-24) were invalid, “Sybase cannot prevail on those claims.”

For those counting along at home — the patent only has 24 claims in total.

I have no idea whether Sybase can still cobble together grounds for appeal, or claims under some other patent. But for now, this sounds like a total victory for Vertica.

Edit: I’ve now seen a PDF of a filing suggesting the grounds under which Sybase will appeal. Basically, it alleges that the judge erred in defining a “page” of data too narrowly. Note that if Sybase prevails on appeal on that point, Vertica has a bunch of other defenses that haven’t been litigated yet. It further seems that Sybase may have recently filed another patent case against Vertica, in a different venue, based on a different patent.

One annoying blog troll excepted, is anybody surprised at this outcome?

Vertica offers a post on its 3.5 release, with a riff on the popular theme “We’ve fixed some weaknesses in our prior versions that we didn’t previously say we had.” More important, Vertica is pretty clear on the virtues of its hybrid columnar architecture.

Speaking of which — Oracle is going true hybrid columnar as well. I don’t have details or timing, however.

Dave Kellogg of Mark Logic wrote in to amusedly point out  Oracle’s anti-MarkLogic collateral. The very first charge Oracle levies is that MarkLogic goes beyond the emerging XQuery standard to add additional functionality. Considering Oracle’s approach to SQL standards, I tend to share Dave’s amusement.

Bill Conniff of Xponent LLC wrote in to tell of a vastly cheaper and less functional approach to XML management, apparently geared to looking at very large XML files one at a time.

Cayuga is a Cornell research project in complex event processing (CEP). There’s a Cayuga academic home page, a Sourceforge page for some open source Cayuga CEP code, and so on. Minsheng Hong, writing from a Vertica email address, tipped me off some months ago. The basic idea seems to be to do lots of queries very quickly, rather than a smaller number of queries over and over again. Whether this is an advance in anything but open-sourceness over Apama or Aleri I couldn’t say, but I do think it’s a different focus than that of StreamBase or pre-Aleri Coral8.

And finally, editor Doug Henschen listed his 15 favorite Intelligent Enterprise blog posts of 2009 — four each by Seth Grimes and Doug himself, three by Cindi Howson, two by me,* and one each by Mark Smith and Neil Raden.

*Doug selects up to three posts a month from here to republish.

• Some are in heavy production with Hadoop, and closely engaged with Cloudera. Others are active Hadoop users but are very secretive. Yet others signed up for initial Hadoop training last week.
• Some have Hadoop clusters in the thousands of nodes. Many have Hadoop clusters in the 50-100 node range. Others are just prototyping Hadoop use. And one seems to be “OEMing” a small Hadoop cluster in each piece of equipment sold.
• Many export data from Hadoop to a relational DBMS; many others just leave it in HDFS (Hadoop Distributed File System), e.g. with Hive as the query language, or in exactly one case Jaql.
• Some are household names, in web businesses or otherwise. Others seem to be pretty obscure.
• Industries include financial services, telecom (Asia only, and quite new), bioinformatics (and other research), intelligence, and lots of web and/or advertising/media.
• Application areas mentioned — and these overlap in some cases — include:
  • Log and/or clickstream analysis of various kinds
  • Marketing analytics
  • Machine learning and/or sophisticated data mining
  • Image processing
  • Processing of XML messages
  • Web crawling and/or text processing
  • General archiving, including of relational/tabular data, e.g. for compliance

We went over this list so quickly that we didn’t go into much detail on any one user. But one example that stood out was of an ad serving firm that had an “aggregation pipeline” consisting of 70-80 MapReduce jobs.

I also talked yesterday again w/ Omer Trajman of Vertica, who surprised me by indicating a high single-digit number of Vertica’s customers were in production with Hadoop — i.e., over 10% of Vertica’s production customers. (Vertica recently made its 100th sale, and of course not all those buyers are in production yet.) Vertica/Hadoop usage seems to have started in Vertica’s financial services stronghold — specifically in financial trading — with web analytics and the like coming on afterwards. Based on current prototyping efforts, Omer expects bioinformatics to be the third production market for Vertica/Hadoop, with telecommunications coming in fourth.

Unsurprisingly, the general Vertica/Hadoop usage model seems to be:

• Do something to the data in Hadoop
• Dump it into Vertica to be queried

What I did find surprising is that the data often isn’t reduced by this analysis, but rather exploded in size. E.g., a complete store of mortgage trading data might be a few terabytes in size, but Hadoop-based post processing can increase that by 1 or 2 orders of magnitude. (Analogies to the importance and magnitude of “cooked” data in scientific data processing come to mind.)

And finally, I talked to Aster a few days ago about the usage of its nCluster/Hadoop connector. Aster characterized Aster/Hadoop users’ Hadoop usage as being of the batch/ETL variety, which is the classic use case one concedes to Hadoop even if one believes that MapReduce should commonly be done right in the DBMS.

Related link

• An August, 2008 round-up of MapReduce applications.

*Guys like Greg Rahn and Kevin Closson are huge assets to Oracle, which is absurdly and self-defeatingly unhelpful through conventional public/analyst relations channels.

Six key tweets by Greg Rahn said:

I think the HCC 10x compression is a slideware (common) number. Personally I’ve seen it in the 12-17x range on customer data…

This was on a dimensional model. Can’t speak to the specific industry. I do believe Oracle is working on getting industry #s.

As far as I know, Exadata HCC uses a superset of compression algorithms that the commonly known column stores use…

…and it doesn’t require the compression type be in the DDL like Vertica or ParAccel. It figures out the best algo to apply.

The compression number I quoted is sizeof(uncompressed)/sizeof(hcc compressed). No indexes were used in this case.

Exadata HCC is applicable for bulk loaded (fact table) data, so a significant portion (size wise) of most DWs.

Summing up, that seems to say:

• Oracle claims 12-17X compression on a kind of data similar to that on which Vertica — which also uses 10X as a single-point overall compression marketing estimate where needed — claims 20X.
• Oracle selects compression algorithms automagically.
• Oracle’s compression doesn’t quite apply to all the data. Actually, this may be more of an issue for the caching benefits of compression than for the I/O or disk storage gains. (If you join a retail transaction fact table to a customer dimension table, and you have a lot of customers, fitting the uncompressed customer table into RAM could be problematic.)

Omer and I happened to have a call scheduled to discuss MapReduce yesterday evening, but wound up using most of the time to talk about Vertica’s compression and physical layout features instead. Highlights included:

• Greg, like many Vertica competitors, was wrong about Vertica requiring manual, low-level DDL (Data Description Language) for — well, for much of anything. Vertica does all that automatically, at least in theory, and suggests that in real life you can indeed often get by without manual intervention.
• Vertica can do trickle feeds into its compressed columnar storage. Greg seemed to suggest Oracle Exadata can not. (However, I won’t be surprised if, when his comments are expanded to more than 140 characters, he winds up saying the opposite. )
• Omer characterized the lowest latency with which you can get data into Vertica and have it be available for query as “seconds”, vs. “minutes” for other columnar vendors.
• Vertica recommends often keeping multiple copies of a column, for high availability and/or performance. This is not directly reflected in compression estimates. In particular, if you’re going to keep redundant copies of data for data-safety reasons anyway, Vertica recommends that you:
  • Run queries against more than one copy of the data, for performance/throughput.
  • Store different copies of the columns in different sort orders — e.g., according to different likely join keys — so that the copies are optimized for performance on different classes of queries.
• Vertica doesn’t have indexes.
• Vertica sorts columns on ingest. This sorting is, of course, commonly based on attributes from columns other than the one being sorted. Even so, Omer maintains that sorting helps compression, because of the correlation between columns. Examples (and I didn’t get these all from him) might include:
  • City/postal code
  • Customer_ID/store location
  • Customer_ID/product_ID
  • Product_ID/price
• Vertica, based on the recent introduction of FlexStore, has an ILM (Information Lifecycle Management) story much like Sybase IQ’s. E.g., you can keep different data ranges for different columns on fast storage, while the rest of the data is relegated to slower/cheaper equipment.

• Vertica/MapReduce integration, which I’m covering in a separate post
• A new storage architecture called Vertica FlexStore, which seems to boil down essentially to three things:
  • A sort of row/column hybridization — Vertica would probably prefer to call it something like a column clustering feature — that I’m also covering in a separate post.
  • The beginnings of a multi-temperature capability, somewhat akin to Teradata Virtual Storage.
  • Enhancements to Vertica’s WOS (Write-Optimized Store, the in-memory part of Vertica that first receives updates). I don’t understand WOS architecture well enough to write about that yet.
• Load-balancing, to route queries evenly among Vertica nodes — probably just round-robin — rather than having them just be processed by whichever node happens to receive them.

And Vertica 3.5 surely includes some lesser features as well.

Like Teradata Virtual Storage, Vertica FlexStore wants to partition data among different parts of storage automatically. But, unlike Teradata’s technology, it will grudgingly let you do it by hand if you insist. Since Vertica installations seem to generally have only one kind of disk each — and that kind spinning rather than solid-state — early tests concentrate on allocating data among the inner and outer tracks of a disk. Vertica said that one typically gets 80% of the benefit by dividing the data into just two partitions, and 90% of the benefit if one divides it into three. However, I don’t recall getting a clear estimate of just how large that benefit is.

• Analytic queries commonly return only a fraction of all possible columns.
• Only returning the columns needed
  • Saves I/O
  • Saves cache space
  • Reduces processing
  • Facilitates compression
• Presumably all those row-based MPP vendors just went row-based because they had a fine row-based DBMS (usually but not always PostgreSQL) to build on.

Pushbacks to this argument from row-based vendors include:

• Yes, but it’s harder to update a column store
• Yes, but there are more steps to retrieving a bunch of columns than there are to retrieving the same information from row stores

plus generous dollops of:

• We’re doing just fine, thank you
• We’re not seeing column stores much in the marketplace
• Don’t believe all that academic hype
• Column stores reek of elderberries, and are powered by hamster wheels

(OK, I made that last one up, but I do hear the other claims frequently.)

However, there are at least two ways in which row- and column-stores are beginning to come together. First, there are lots of rumors about row-store vendors bringing out column-store options, even beyond the recent Ingres/VectorWise announcement. (But anything I may know about same beyond noticing the rumors fly by is surely under NDA.) Second, column-store vendors Vertica and VectorWise are bringing out a kind of row/column hybrid storage option.

Vertica 3.5 introduces what Vertica calls “FlexStore.” A key part of FlexStore is the ability to store data not just in pure columnar format, but also to group columns together in what amounts to sub-rows. This is advantageous when data is retrieved together and, I presume, when it is updated. There’s a tradeoff in giving up column stores’ compression advantages, however, and use of this feature is not recommended for columns that are frequently retrieved independently. Vertica also notes that since it typically uses 1 megabyte block sizes, any table smaller than that shouldn’t be broken into columns at all.

VectorWise, of course, doesn’t have a product right now, but has gotten a bunch of recent publicity around the column-store product it plans to ship via its partner Ingres in 2010. When I asked Peter Boncz about row/column hybridization inside VectorWise (not federating between Ingres and VectorWise, but rather truly within VectorWise), he said one of the storage options was PAX, and pointed me at a 2001 paper by a group of academics that includes the ubiquitous Dave Dewitt. PAX turns out to stand, in creative spelling, for Partition Attributes Across.

The PAX idea is to store as many rows of data as can fit into a block, but within the block store them in columns. This preserves some of the compression and cache-efficiency benefits of column stores, while also bringing back whole rows in a single step. (I think Vertica’s FlexStore does something similar to this, but I’m not sure.)

Further confusing things, Peter Boncz of VectorWise told me VectorWise can support “any hybrid” of columnar storage and PAX.

Bottom line: The distinction between row- and column-stores isn’t going to go away any time soon, but it is at least beginning to blur a bit.

• By “integrating Vertica and MapReduce,” Vertica means “integrating Vertica and Hadoop.”
• Vertica’s Hadoop integration is based on Cloudera’s DBInputFormat.
• Omer called out for me several features of Vertica’s Hadoop integration that didn’t just come from Cloudera, namely:
  • Cloudera’s DBInputFormat assumes the database runs on a single computer, or a single head node of an MPP system. Vertica’s technology, however, runs on peer parallel nodes with no head, and so Vertica adapted the DBInputFormat technology accordingly.
  • Vertica lets you push down Map functions to the database. Omer reports a roughly even division among users and prospects between those who want to do this and ones who don’t.
  • Vertica lets you do Reduce functions (or Map functions, if you don’t push them down to the database) on a separate cluster than you run the database software. Vertica asserts that its customers and prospects all want to do this. Right here is the big difference between Vertica’s MapReduce integration and Aster’s or Greenplum’s. (Aster would also say that Vertica’s weaker MapReduce/SQL programming integration is a big difference as well.)
  • Indeed, Vertica lets you Reduce into a different DBMS than Vertica, if you choose.
  • Vertica gives you flexibility on the size of the Map and Reduce clusters. Omer agreed with me when I said there were some limits on how fast one can add or subtract nodes in a Vertica grid, because there’s data redistribution involved. But one can add/change/delete Hadoop clusters extremely quickly.

Apparently, the use cases for Vertica/Hadoop integration to date lie in algorithmic trading and two kinds of web analytics. Specifically:

• One or more Vertica customers are using MapReduce in production to do relatively simple transforms of web log data
• Vertica customers are experimenting with — but have not yet put into production — more sophisticated pattern analysis of web log data.
• Financial services customers are using MapReduce for a lot of experimentation in discovering new algorithms. The idea is that DBMS/MapReduce integration offers rapid prototyping of algorithmic ideas. Those that pan out are then reimplemented for production, presumably in some kind of CEP (Complex Event Processing) system. These users seem to be ones that are pushing down a lot of Map functions to the Vertica DBMS.

By the way, Vertica is based on C-Store, the Ph.D. thesis project of Daniel Abadi, who recently wrote:

To me, it is far more efficient from a performance and a “green” perspective to push the computation to the data. Hence, I am not a fan of decoupling the compute grid and the data grid.

Not coincidentally, Daniel also recently wrote that

If the VectorWise/Ingres solution does get released open source, I believe they will be an excellent column-store storage engine for HadoopDB. I have already requested an academic preview edition of their software to play with.

The VectorWise guys also told me they are looking forward to seeing how the two projects work together.