]]> http://www.dbms2.com/2009/09/30/facts-and-rumors/feed/ 2 http://www.dbms2.com/2009/06/08/the-future-of-data-marts/ http://www.dbms2.com/2009/06/08/the-future-of-data-marts/#comments Mon, 08 Jun 2009 08:25:07 +0000 Curt Monash http://www.dbms2.com/?p=805 Greenplum is announcing today a long-term vision, under the name Enterprise Data Cloud (EDC). Key observations around the concept — mixing mine and Greenplum’s together — include:

• Data marts aren’t just for performance (or price/performance). They also exist to give individual analysts or small teams control of their analytic destiny.
• Thus, it would be really cool if business users could have their own analytic “sandboxes” — virtual or physical analytic databases that they can manipulate without breaking anything else.
• In any case, business users want to analyze data when they want to analyze it. It is often unwise to ask business users to postpone analysis until after an enterprise data model can be extended to fully incorporate the new data they want to look at.
• Whether or not you agree with that, it’s an empirical fact that enterprises have many legacy data marts (or even, especially due to M&A, multiple legacy data warehouses). Similarly, it’s an empirical fact that many business users have the clout to order up new data marts as well.
• Consolidating data marts onto one common technological platform has important benefits.

In essence, Greenplum is pitching the story:

• Thesis: Enterprise Data Warehouses (EDWs)
• Antithesis: Data Warehouse Appliances
• Synthesis: Greenplum’s Enterprise Data Cloud vision

When put that starkly, it’s overstated, not least because

Specialized Analytic DBMS != Data Warehouse Appliance

But basically it makes sense, for two main reasons:

• Analysis is performed on all sorts of novel data, from sources far beyond an enterprise’s core transactions. This data neither has to fit nor particularly benefits from being tightly fitted into the core enterprise data model. Requiring it to do so is just an unnecessary and painful bureaucratic delay.
• On the other hand, consolidation can be a good idea even when systems don’t particularly interoperate. Data marts, which commonly do in part interoperate with central data stores, have all the more reason to be consolidated onto a central technology platform/stack.

Of course, the EDC vision isn’t quite as new or differentiated as Greenplum ideally would wish one to believe.

• To a first approximation, EDC sounds a lot like what eBay has already built on Teradata equipment.
• Greenplum’s EDC vision also sounds a lot like what Stuart Frost was talking about at DATAllegro, what Dell was planning to build on DATAllegro equipment, and what Stuart continues to talk about now that he’s been acquired into Microsoft.
• Something like EDC can also be presumed to be implicit in the strategies of the other one-size-fits-all vendors — i.e., Oracle and IBM.
• Greenplum has only implemented a little more of the EDC vision so far than have other firms, unless you give it credit for being cheap/fast/MPP/running on commodity hardware, but deny that credit to Teradata (specialized hardware, and not cheap in its most popular configurations), Oracle (ditto for Exadata), IBM (also not cheap), or Microsoft/DATAllegro (not released yet).
• Specifically: In Greenplum Release 3.3, which is being announced today, Greenplum is introducing the (enhanced?) ability for data marts to be spun out as a background operation, while the database otherwise remains functional. As of 3.3, spinning out a data mart is a command-line operation. But in Release 3.4, Greenplum plans to offer a web-based interface for same, at which point the “self-service data mart creation” discussion will become operative. Otherwise, EDC is a roadmap/vision/statement-of-direction much more than it is a fully-baked technical project.

One particular source of potential confusion is Greenplum’s emphasis on the buzzphrase self-service (data mart). This seems to be a conflation of two related concepts:

• End users should be able to create new data marts themselves. Strictly speaking, I view this ability as useless at most enterprises, and important at very few, because of logistical issues. (Who gives the permissions? Who decides which hardware is used?) That said, useless “end user” tools often wind up being important productivity aids for IT professionals, and this kind of “self-service” would surely be another example.  Edit: Hmm. Doug Henschen inspired me to think that over again, and I’m beginning to soften. Suppose users could order up the data mart they want, perhaps test it at a very low processing priority (if they choose), and then send the completed request to IT for approval and provisioning. That would have some value.
• End users should be able to manage data marts themselves, once created. That’s a great idea, full of agility and don’t-make-IT-a-roadblock goodness. Data miners and similar analytic professionals commonly have the technical ability to manage a simple database, and should be allowed to do so if it’s ensured that they don’t break anything for anybody else.

One thing that’s needed for this technology to come to full fruition is sophisticated data movement and synchronization. Ideally, some tables in a data mart could be virtual — views against a central database. But others would be physically recopied from the center, with all the ETL/ELT/ETLT/replication issues that entails. Meanwhile, it’s not obvious that the ideal architecture is a simpleminded hub-spoke — perhaps one should be able to spin data marts out of other marts, perhaps at least somewhat reducing the proliferation of tables and the recopying of data. And it should be easy for administrators to change deployment strategies, e.g. by starting a table out as a view and changing over to making it a physical copy as usage profiles change.

Oliver Ratzesberger of eBay also argues that workload management — not a current Greenplum strength — can be crucial. For example, if the CEO wants the CFO to get her an answer TODAY, the fastest approach may be to create an entirely virtual data mart, with very favorable SLAs (Service Level Agreements). More generally, if you’re setting up dozens of marts that contain views of the central database, sophisticated SLA management can be essential. There’s a big virtualization opportunity here — but virtualization requires a lot of system management infrastructure.

Related links

• My recent post on reinventing business intelligence
• Greenplum adviser Joe Hellerstein’s pitch for agile data warehousing
• Charlie Bachman’s “private database” idea, which never went anywhere (pp. 138-139)
• Greenplum’s EDC and Release 3.3 press releases
• An interview with some of Greenplum co-founder Scott Yara’s own words

Microsoft purchased DATAllegro for $275 million

Technically, that needn’t shut down the rumor mill altogether, since given the way deals are structured and reported, it’s unlikely that Microsoft actually cut checks to DATAllegro stockholders in the aggregate amount of $275 million promptly after the close of the acquisition.

Still, it’s a data point of some weight.

Hat tip to Mark Myers.

The number of DATAllegro production references is expected to double imminently, from one to two. Few will be surprised at the identity of the second reference. I imagine the number will then stay at two, as DATAllegro technology is no longer being sold, and the third known production user has never been reputed to be particularly pleased with it.

Dell did indeed disclose at TDWI that it was a large DATAllegro user, notwithstanding that Dell is a huge Teradata user as well.  No doubt, Dell is gearing up to be a big user of Madison too.

Also at TDWI, I talked with some former DATAllegro employees who now work for rival vendors. None thinks DATAllegro has more than three customers.  Neither do I.

So I might as well post sound bites in advance. Here goes:

• With the DATAllegro acquisition, Microsoft leapfrogged Oracle. But with Exadata, Oracle leapfrogged Microsoft back. Exadata is actually shipping.
• There’s no assurance that the first DATAllegro/Microsoft release will inherit SQL Server’s level of concurrency. After all, DATAllegro/Ingres wasn’t as concurrent as plain Ingres.
• Porting DATAllegro from Ingres to SQL Server is likely to be straightforward. If they screw up it will be because they tried to do too much else at the same time, not because the basic port failed.
• Porting DATAllegro from Linux to Windows should also be OK. DATAllegro doesn’t stress the operating system in the areas where Windows remains weak.
• Earlier this year, DATAllegro had exactly one customer known to be in production, but I’ve spoken with that one. It’s TEOCO, which has a multi-hundred terabyte DATAllegro installation. TEOCO is a very price-oriented buyer.
• DATAllegro reports that two more customers are in production with large systems now. Neither of those is believed by industry sources to be especially in love with DATAllegro. Otherwise, nobody seems able and willing to identify other DATAllegro customers.

I’m going to be pretty busy Monday anyway. Linda is having a bit of oral surgery. And if I get back from that in time, I have calls set up with a couple of clients.

My resolve to eschew scathing sarcasm is being sorely tested tonight. The lastest trial is my discovery that nobody thought to so much as email me a press release, let alone brief me, on Microsoft’s announcement of a timetable for DATAllegro/SQL Server integration. Per Ina Fried — with a hat tip to anonymous commenter L.J. — Microsoft says:

The final version of that product is slated for the first half of 2010, though Microsoft said it will begin giving customers and partners access to early “community technology preview” releases within the next 12 months.

• Aster Data
• Calpont
• DATAllegro
• Greenplum
• Infobright
• Netezza
• ParAccel
• Teradata
• Vertica

All of those are Monash Advantage members.

If you care about all this, you may also be interested in the rest of my standards and disclosures.

As a by-the-by to other discussions, DATAllegro Stuart Frost undertook to respond in part himself. Specifically, he gave me two names of two other happy customers that are or imminently will be running DATAllegro against 100+ terabytes of user data. The names are confidential, but both are enterprises whose names have long been linked to DATAllegro’s, and both are ones about which there’s been some doubt as to their DATAllegro relationship in the past. Now, I haven’t actually checked those references myself, but I’m guessing that if somebody else tries to at this point, they’ll actually find happy users.

Obviously, three’s not a very large number for an overall customer base. Indeed, one can find more than that in the men’s room at a Netezza user conference. But three customers at the 100+ terabyte level is a more serious accomplishment. Teradata is probably the only rival that’s well ahead of that figure, and I’m actually not 100% sure that anybody else has even matched it. (But then, to be technical, neither quite yet has DATAllegro itself.)

I’ve been a software industry analyst since 1981, covering DBMS throughout that period. Perhaps I’m getting senile, but I can recall only 1 software patent case in those 27 years that wound up having any material effect on enterprise software users — Marcam/Ross Systems, about process manufacturing ERP software. Business Objects/Cognos doesn’t count; Business Objects enjoyed the cash, but Cognos easily afforded the payments. Lotus/Borland about Quattro Pro’s compatibility with 1-2-3 doesn’t count either; the issue there wasn’t patents.

So I haven’t studied the patents and cases in question seriously myself, since I think it’s very unlikely the trouble to do so will prove worthwhile. That said, DATAllegro CEO Stuart Frost told me today that Cory Jardin’s patent was something he thought was invalid before it ever was submitted, because of prior art, and that the application was formally submitted without reference to prior art. (Bad idea if you want the patent to hold up to reexamination when you try to enforce it later.) He further claims that DATAllegro doesn’t come close to infringing anyway. Needless to say, he also rejects the allegation that his own patents or applications in the area are for similar claims to those Jardin asserts.

As for the Sybase case — I don’t know which specific patents are alleged to be infringed. That said, US Patents 5794229 and 5918225 (issued in 1998 and 1999 respectively), both read in their first claim as if they pretty much patent the idea of columnar SQL DBMS with compressed data, especially dictionary/token compression. The “SQL” part is important, because otherwise they’d be blown out immediately by all sorts of prior art, e.g. from Essbase and Verity, or even Excel. Still, given the prevalence of ODBC access even in the 1990s, I’m hard-pressed to imagine how those patents could have any significant validity.

Also, the Sybase case was filed in Texas, which seems to be an oddly irrelevant venue. I wonder what their motive, or even excuse, for that choice was.

Related links

• Eric Lai’s article on the DATAllegro suit, with further basic links
• Stuart Frost’s short blog post responding to the suit
• What appears to be docket information for the Sybase/Vertica suit

One linking screwup is that I previously forgot to mention that — in addition to the numerous posts here — I also made several DATAllegro/Microsoft-related posts on my Network World blog A World of Bytes.  They include:

• A quick primer on specialty data warehouse technologies (nothing new there if you’re a regular reader of this blog)
• An overview of why this announcement is important.
• Some quick notes on Infiniband, scale-out, and 10GbE.

Also, I’d like to correct an error that’s crept into some of the coverage of the deal, for example InformationWeek’s (quoting Gartner).  Folks, there is no way on Earth that Microsoft is going to continue DATAllegro’s strategy of reselling a single or very limited range of hardware choices as the only way to get its high-end data warehousing software.  Rather, Microsoft is going to have multiple “reference architectures,” covering many hardware, storage, and networking options.  I.e., Microsoft will pursue the data warehouse packaging strategy Oracle is following today, but will do so with much better technology.