When you sell a row-based DBMS, as Netezza and IBM do, there are a couple of approaches you can take to compression. First, you can compress the blocks of rows that your DBMS naturally stores. Second, you can compress the data in a column-aware way. Both Netezza and IBM have chosen completely column-oriented compression, with no block-based techniques entering the picture to my knowledge. But that’s about as far as the similarity between Netezza and IBM compression goes. 

IBM’s basic DB2 compression strategy is remarkably simple. In every table (not column) – or in each range partition in a range-partitioned table — the 4096 most common* values are identified; these are all encoded into 12-bit strings. And that’s that. This has been happening since DB2 9.1, released 4 ½ years ago. DB2’s compression persists through logs, buffer pools (i.e., RAM cache), and so on. In DB2 9.7, the most recent release, IBM extended the use of the compression to a few areas it hadn’t stretched before, such as log-based replication, native XML, or CLOBs (Character Large OBjects) that happen not to be too big.

*Actually, I’d presume it’s not exactly the “most common”; there surely is some minimum length of a value to be encoded, or some bias toward length. Also, the determination of what to encode is probably a little imprecise. E.g., I forgot to ask whether the choice of values ever changes as data got updated.

The sophisticated part of DB2’s simple compression strategy is its breadth of applicability; DB2 compression can apply to:

• Values in columns (numeric, character, whatever)
• Substrings of values in columns
• Groups of columns (e.g., city/state/zip code)

Except for the 4096 values limit, that sounds at least as flexible as the Rainstor/Clearpace compression approach.

Netezza, unlike IBM, takes a grab-bag approach to compression – try out a bunch of techniques, see which work best, and incorporate those in the product. Netezza first introduced compression a couple of years ago, for numeric columns only, especially integer.  Techniques  used in Netezza numeric compression include but are not limited to:

• Delta compression, wherein you store the increment between a value and its predecessor rather than a whole new value.
• Ways of indicating that a value or increment was just the same as in the row before.

This was via something called Compress Engine,* now being renamed to Compress Engine 1. Netezza’s new Compress Engine 2 improves on what Netezza did in Compress Engine 1 for numeric data, most notably by trimming away excess field length. (Netezza says it got 28% better compression on a test data set with almost no character strings, primarily from that enhancement.) Further, Netezza Compress Engine 2 adds new compression techniques, allowing it to handle VARCHAR – i.e. character strings — as well.

*Fortunately, the original name or at least description of “Compiled Tables” is retreating ever more from view.

Netezza’s Compress Engine 2 has two ways to compress character fields/text strings – prefix compression and Huffman coding. By way of contrast, Netezza tested suffix compression and decided it wasn’t beneficial enough to bother messing with.

• The idea behind prefix compression is that if two strings start with the same characters, for the second one you only have to record the part that’s different. Prefix compression has a lot of the same merits as delta compression; like delta compression, it works best on sorted columns. (An example of where prefix compression makes obvious sense is URLs, which tend to all start in similar ways.)
• In Netezza’s version of Huffman coding, the alphabet is encoded symbol-by-symbol, with more common characters getting codes of shorter length. These codes are chosen on a column-by-column basis. (I presume the “/” character gets shorter code in a URL column than it would, for example, in one that stored addresses.)

While I didn’t ask explicitly, it seems pretty obvious that Compress Engine 2’s functionality is a strict superset of Compress Engine 1’s. Netezza is going to run Compress Engines 1 and 2 side by side, but expects pages to move from Compress Engine 1’s purview to Compress Engine 2’s as part of the new “table grooming” process.

Related links

• IBM kindly permitted me to post some of its slides in the area of compression
• Microsoft SQL Server seems to rely on prefix and dictionary compression

The main new activity I know of has been in the open source SciDB project.  

• A company called Zetics has been started to commercialize SciDB. As of now, the entire staff seems to be CEO Marilyn Matz, techie Paul Brown, and part of Mike Stonebraker. Marilyn says Zetics has some venture capital, but even under NDA didn’t tell me who it was from. Zetics does not have its own web site.
• Marilyn tells me there are 20-25 contributors to SciDB, led by Paul Brown and Mike Stonebraker. Brown is full-time. Persistent Systems has been donating the efforts of a few of its employees. Some LSST folks have been doing SciDB work backed by grant money. Most or all of the rest seem to be purer volunteers. Some Russians have been particularly active.
• Release 0.5 of SciDB is expected in June. Release 1.0 is expected in September. This is a rewrite; prior demo code has been scrapped. Perhaps not coincidentally, it’s also a small slip from prior project plans.
• The array data model is an example of what’s being implemented first. (Duh — you can’t have a DBMS without a data model.) Support for uncertainty is an example of what’s been deferred until later.
• As has been clear since XLDB3 last August, one major target market for SciDB is genomic research.
• It’s obvious that the oil and gas industry, with all its geospatial data, should be interested in SciDB. But there’s not much activity in that regard; outreach is evidently needed. If you can think of somebody in that sector (or anywhere else) who should be alerted to SciDB, please ping them.
• Interest from web analytics users in SciDB seems to have receded a bit from the days when eBay almost funded the project.

In other scientific data management news,

• Microsoft put out a book called The Fourth Paradigm on scientific database management. The whole thing can be downloaded, very officially, as a giant PDF. I think it’s worth skimming. I don’t think it’s worth actually reading. (I did read it.)
• XLDB4 will be at Stanford October 5-7. Unlike prior XLDBs, it will have an open (i.e., no invitation required) part.

Finally, you surely are aware of the whole “Climategate” mess, in which major climate researchers’ email was hacked and many unkind conclusions were drawn. Well, one of the most technical parts of the disclosure was in a long series of Read Me files, in which an unfortunate programmer lamented about the difficulty of reconstructing published results from files at hand. These turned out to illustrate a classic problem that SciDB or alternatives are meant to solve:

• Raw data was impossible to use without various adjustments to regularize it (the word “regridding” comes up a lot, for example). Massaging was needed before analytics could be done on it.
• The raw data was thrown out or lost, and could not be reconstructed (why they couldn’t have asked the suppliers of the data to give it to them again was unclear in this case, since it wasn’t original experimental data).
• It was thus impossible to massage the data in any new or improved way.

• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

]]> http://www.dbms2.com/2009/09/30/facts-and-rumors/feed/ 2 http://www.dbms2.com/2009/09/11/xkoto-gridscale-highlights/ http://www.dbms2.com/2009/09/11/xkoto-gridscale-highlights/#comments Fri, 11 Sep 2009 18:36:03 +0000 Curt Monash http://www.dbms2.com/?p=881 I talked yesterday with cofounders Albert Lee and Ariff Kassam of Xkoto. Highlights included:

• Xkoto sells Gridscale, a clustering server for DB2 and, more recently, MS SQL Server.
• Xkoto Gridscale runs on a separate box, between the application and the database servers. This box is typically smaller and cheaper than the database server boxes.
• Xkoto most typically sells Gridscale into environments where there already are three database servers — one to do work, one for hot standby, and one for remote disaster recovery.
• In such environments, Gridscale’s big benefit is that you can distribute the query workload among all three servers. Xkoto believes this big performance increase is the reason customers don’t get much past 3 database servers under Xkoto (they didn’t seem quite sure as to whether the all-time record was 4 or 5). Note that even if a remote server is a little too far away for OLTP query response, it can work fine for reporting.
• Of course, if you don’t already have high/”continuous” availability and/or disaster recovery, then Xkoto would say those are core benefits of Gridscale as well.
• Gridscale sends transactions (or just SQL statements?) to all servers in the cluster. Once any of them responds affirmatively, that update is reflected in queries. Gridscale maintains a small query log to make sure it gets the other database copies in sync. It also tries to make sure that queries always go to the most current copy of the database. (I didn’t ask what happens if Server A executes Transaction T but not U, while Server B executes Transaction U and not T — but that does seem like something of an edge case.).
• Xkoto spun out of Halcyon Monitoring in 2006, starting with DB2 support. Microsoft SQL Server support was introduced in 2008.
• Xkoto likes its partnerships with IBM and Microsoft. For example, IBM provides Level 1 and 2 support for Gridscale itself. Due in large part to this partnership strategy, Xkoto says it has no plans to support DBMS beyond DB2 and SQL Server.
• Instead, Xkoto is pursuing partnerships with large application vendors and so on. (The figure “about 10″ was mentioned.) I gather the idea is to make sure that neither the application support folks nor the app itself freak out from the fact that the app isn’t exactly talking to the DBMS any more.
• Xkoto has done lab tests suggesting Gridscale offers near-linear scalability (in terms of SQL Server database throughput) on a query-only workload up to 10 servers.
• I gather that Xkoto and IBM have demos suggesting it’s a fine idea to have your disaster recovery server be in the Amazon cloud, but they haven’t yet made any sales based on that — er, based on that premise.
• Gridscale pricing is measured in the same metrics as DB2 or SQL Server pricing, and in each case is around 1/3 what database pricing would be on the same box (I’m guessing that’s for enterprise additions without add-ons, but I didn’t probe). Specifically, Gridscale charges $12K per 100 PVUs for the DB2 edition, and $12K per socket for running with Microsoft SQL Server.
• Gridscale typically runs on smaller boxes than the databases it talks to.
• Xkoto has about 35 revenue-recognized customers. Most are on DB2, the first environment Gridscale supported.
• Average Gridscale selling prices are $180K on DB2, and $40-50K in the early going for SQL Server.
• Xkoto has about 40 full-time employees, with engineering in Toronto and business operations in Waltham.

Microsoft Madison availability remains scheduled for H1 2010. Nothing new there. Tangible progress includes a few customer commitments of various sorts, including one outright planned purchase (due to some internal customer considerations around using up a budget). At the moment various Microsoft Madison technology “previews” are going on, which seem to amount to proofs-of-concept, that:

• Start with actual customer data (some from Microsoft, some from outside)
• Generate larger synthesized data sets based on those (database size seems to be 10-100 TB)
• Run in Microsoft data centers or “technology centers”, rather than on customer premises.

The basic Microsoft Madison product distribution strategy seems to be:

• Microsoft specifies configurations with different brands of hardware.
• The respective hardware vendors sell and deliver those configurations, with Madison software pre-installed.
• Actual software licenses are invoiced however makes sense, which in most cases may well be directly by Microsoft as part of a wider enterprise relationship.

Most of the usual-suspect big name hardware and storage vendors seem to be involved.

Microsoft Madison is focused on “high-end” data warehousing, with Stuart candidly saying that everybody at Microsoft seems to have a different definition of what “high-end” means. In practice, this “high end” probably will be whatever conventional SQL Server doesn’t do a good job on — e.g., >5 terabytes, or even smaller in table-scan-oriented workloads.

Microsoft Madison seems further focused on being the “hub” to SQL Server data marts, with Stuart citing a survey saying Microsoft SQL Server has 44% market share in data marts when counting by unit. When I pressed for technical strategy as to how the data would be moved and synchronized between the hub and spokes, details were vague. Obviously, Microsoft still has considerable work to do in this regard, whether in articulating strategy or in actual product development. The same goes double for ease-of-data-mart creation ala Greenplum’s Enterprise Data Cloud messaging.

While Madison is a future, Stuart says Microsoft SQL Server Fast Track is a “huge hit.” He knows of 20 sales, and estimates the total number is in the 100s. The disparity is explained by the fact that that Fast Track comprises a set of recommended hardware configurations, rather than an identifiable Microsoft software invoice line item. The distribution model for Microsoft SQL Server Fast Track seems to be similar to that planned for Madison, with hardware partners such as HP making many of the sales.

Sales of all these Microsoft SQL Server data warehousing products seem to be focused on the Microsoft SQL Server installed base. No surprise there either.

One of my favorite pages on the Monash Research website is the list of many current and a few notable past customers. (Another favorite page is the one for testimonials.) For a variety of reasons, I won’t undertake to be more precise about my current customer list than that. But I don’t think it would hurt anything to list the analytic/data warehouse DBMS/appliance specialists in the group. They are:

• Aster Data
• Greenplum
• Infobright
• Kickfire
• Kognitio
• Microsoft
• Netezza (my biggest client this year, probably, because of all the Enzee Universe appearances)
• Sybase
• Teradata
• Vertica
• Attivio, which may or may not be construed as being in the analytic DBMS business
• Clearpace, ditto

All of those are Monash Advantage members.

If you care about all this, you may also be interested in the rest of my standards and disclosures.

• Data marts aren’t just for performance (or price/performance). They also exist to give individual analysts or small teams control of their analytic destiny.
• Thus, it would be really cool if business users could have their own analytic “sandboxes” — virtual or physical analytic databases that they can manipulate without breaking anything else.
• In any case, business users want to analyze data when they want to analyze it. It is often unwise to ask business users to postpone analysis until after an enterprise data model can be extended to fully incorporate the new data they want to look at.
• Whether or not you agree with that, it’s an empirical fact that enterprises have many legacy data marts (or even, especially due to M&A, multiple legacy data warehouses). Similarly, it’s an empirical fact that many business users have the clout to order up new data marts as well.
• Consolidating data marts onto one common technological platform has important benefits.

In essence, Greenplum is pitching the story:

• Thesis: Enterprise Data Warehouses (EDWs)
• Antithesis: Data Warehouse Appliances
• Synthesis: Greenplum’s Enterprise Data Cloud vision

When put that starkly, it’s overstated, not least because

Specialized Analytic DBMS != Data Warehouse Appliance

But basically it makes sense, for two main reasons:

• Analysis is performed on all sorts of novel data, from sources far beyond an enterprise’s core transactions. This data neither has to fit nor particularly benefits from being tightly fitted into the core enterprise data model. Requiring it to do so is just an unnecessary and painful bureaucratic delay.
• On the other hand, consolidation can be a good idea even when systems don’t particularly interoperate. Data marts, which commonly do in part interoperate with central data stores, have all the more reason to be consolidated onto a central technology platform/stack.

Of course, the EDC vision isn’t quite as new or differentiated as Greenplum ideally would wish one to believe.

• To a first approximation, EDC sounds a lot like what eBay has already built on Teradata equipment.
• Greenplum’s EDC vision also sounds a lot like what Stuart Frost was talking about at DATAllegro, what Dell was planning to build on DATAllegro equipment, and what Stuart continues to talk about now that he’s been acquired into Microsoft.
• Something like EDC can also be presumed to be implicit in the strategies of the other one-size-fits-all vendors — i.e., Oracle and IBM.
• Greenplum has only implemented a little more of the EDC vision so far than have other firms, unless you give it credit for being cheap/fast/MPP/running on commodity hardware, but deny that credit to Teradata (specialized hardware, and not cheap in its most popular configurations), Oracle (ditto for Exadata), IBM (also not cheap), or Microsoft/DATAllegro (not released yet).
• Specifically: In Greenplum Release 3.3, which is being announced today, Greenplum is introducing the (enhanced?) ability for data marts to be spun out as a background operation, while the database otherwise remains functional. As of 3.3, spinning out a data mart is a command-line operation. But in Release 3.4, Greenplum plans to offer a web-based interface for same, at which point the “self-service data mart creation” discussion will become operative. Otherwise, EDC is a roadmap/vision/statement-of-direction much more than it is a fully-baked technical project.

One particular source of potential confusion is Greenplum’s emphasis on the buzzphrase self-service (data mart). This seems to be a conflation of two related concepts:

• End users should be able to create new data marts themselves. Strictly speaking, I view this ability as useless at most enterprises, and important at very few, because of logistical issues. (Who gives the permissions? Who decides which hardware is used?) That said, useless “end user” tools often wind up being important productivity aids for IT professionals, and this kind of “self-service” would surely be another example.  Edit: Hmm. Doug Henschen inspired me to think that over again, and I’m beginning to soften. Suppose users could order up the data mart they want, perhaps test it at a very low processing priority (if they choose), and then send the completed request to IT for approval and provisioning. That would have some value.
• End users should be able to manage data marts themselves, once created. That’s a great idea, full of agility and don’t-make-IT-a-roadblock goodness. Data miners and similar analytic professionals commonly have the technical ability to manage a simple database, and should be allowed to do so if it’s ensured that they don’t break anything for anybody else.

One thing that’s needed for this technology to come to full fruition is sophisticated data movement and synchronization. Ideally, some tables in a data mart could be virtual — views against a central database. But others would be physically recopied from the center, with all the ETL/ELT/ETLT/replication issues that entails. Meanwhile, it’s not obvious that the ideal architecture is a simpleminded hub-spoke — perhaps one should be able to spin data marts out of other marts, perhaps at least somewhat reducing the proliferation of tables and the recopying of data. And it should be easy for administrators to change deployment strategies, e.g. by starting a table out as a view and changing over to making it a physical copy as usage profiles change.

Oliver Ratzesberger of eBay also argues that workload management — not a current Greenplum strength — can be crucial. For example, if the CEO wants the CFO to get her an answer TODAY, the fastest approach may be to create an entirely virtual data mart, with very favorable SLAs (Service Level Agreements). More generally, if you’re setting up dozens of marts that contain views of the central database, sophisticated SLA management can be essential. There’s a big virtualization opportunity here — but virtualization requires a lot of system management infrastructure.

Related links

• My recent post on reinventing business intelligence
• Greenplum adviser Joe Hellerstein’s pitch for agile data warehousing
• Charlie Bachman’s “private database” idea, which never went anywhere (pp. 138-139)
• Greenplum’s EDC and Release 3.3 press releases
• An interview with some of Greenplum co-founder Scott Yara’s own words

Natural language and classic science fiction

Actually, there’s a pretty well-known example of BI near-perfection — the Star Trek computers, usually voiced by the late Majel Barrett Roddenberry. They didn’t have a big role in the recent movie, which was so fast-paced nobody had time to analyze very much, but were a big part of the Star Trek universe overall. Star Trek’s computers integrated analytics, operations, and authentication, all with a great natural language/voice interface and visual displays. That example is at the heart of a 1998 article on natural language recognition I just re-posted.

As for reality: For decades, dating back at least to Artificial Intelligence Corporation’s Intellect, there have been offerings that provided “natural language” command, control, and query against otherwise fairly ordinary analytic tools. Such efforts have generally fizzled, for reasons outlined at the link above. Wolfram Alpha is the latest try; fortunately for its prospects, natural language is really only a small part of the Wolfram Alpha story.

A second theme has more recently emerged — using text indexing to get at data more flexibly than a relational schema would normally allow, either by searching on data values themselves (stressed by Attivio) or more by searching on the definitions of pre-built reports (the Google OneBox story). SAP’s Explorer is the latest such view, but I find Doug Henschen’s skepticism about SAP Explorer more persuasive than Cindi Howson’s cautiously favorable view. Partly that’s because I know SAP (and Business Objects); partly it’s because of difficulties such as those I already noted.

Flexibility and data exploration

It’s a truism that each generation of dashboard-like technology fails because it’s too inflexible. Users are shown the information that will provide them with the most insight. They appreciate it at first. But eventually it’s old hat, and when they want to do something new, the baked-in data model doesn’t support it.

The latest attempts to overcome this problem lie in two overlapping trends — cool data exploration/visualization tools, and in-memory analytics. Tableau and Spotfire are known more for the former; hot BI vendor QlikTech is known for both. And many vendors — established or otherwise — are going to in-memory OLAP.

Collaboration and communication

The reason I’m finally buckling down and posting on this subject is the announcement of Google Wave, which I think foreshadows a revolution in communication and collaboration technology. Google Wave augurs two primary advances. First, it shows how to make email, instant messaging, microblogging, and so on much more useful. Second, Google Wave could evolve in a way that — finally — makes it truly practical for end-users to set up ad-hoc mini-portals that combine arbitrary URL-possessing resources, exposed to arbitrary workgroups of people.

If and when both of those promises are fulfilled, it will become vastly easier for people to reason together about analytic questions. That may take a little while, as Google Wave obviously wasn’t designed with business intelligence in mind. But whether from Google or from a frightened Microsoft redoubling its SharePoint efforts, there’s hope that we’ll see a leap forward in general collaboration technology. And since BI vendors are doing a generally decent job of exposing queries, charts and so on as portlets, it seems likely that business intelligence will benefit from the collaboration arms race.

That’s important. The first time I heard that reporting was as important for communication as for analytics was from Pilot Software a quarter-century or so ago, and it’s just as true now as it was then. In its first incarnations it probably will be a little too dumb for my tastes, focusing more on mindless reporting and same-old KPIs than on deeper analysis. Still, it’s a move in a good direction.

Other directions

As I said at the beginning, I find it too daunting to try to cover all facets of this subject in one post. So I’ll leave out, at a minimum:

• Data warehousing performance and TCO, which I of course write about extensively
• Complex event/stream processing, which I’ve written quite a bit about too
• Data mining and predictive analytics
• Operational BI

plus some hobby horses you probably don’t want to hear about anyway until I work out a better way of articulating my opinions.

But by all means please comment on what I’ve left out just as vigorously as on what I’ve included. This post is just the first of many to come.

]]> http://www.dbms2.com/2009/05/30/reinventing-business-intelligence/feed/ 15