• Ben says Greenplum wasn’t being shopped, by which he means Greenplum was out raising more capital and the fund-raising was going well.  Note: Half or so of Greenplum’s deals were subscription-priced, so it had weaker cash flow than it would have if it were doing equally well selling perpetual licenses.
• However, joint engineering was also going well with, e.g., Greenplum CTO Luke Lonergan spending time at EMC facilities in Cork, Ireland. And one thing led to another …
• Greenplum has ~ 140 customers, vs. ~65 five quarters ago, 100+ at year-end, and an acquisition rate of 12-15/quarter last fall.
• A typical “small” paying customer for Greenplum starts with 10-20 TB of data.
• Greenplum Chorus isn’t generally available yet, with rollout energy being focused on Greenplum 4.0. Note: As important as it is for overall industry direction, Greenplum Chorus is a product which won’t be a terribly big deal in Release 1 anyway.

Highlights looking forward include: 

• When I challenged him, Ben sounded quite optimistic that Pat Gelsinger will immunize Greenplum against and generally counteract some of EMC’s traditionally stifling bureaucracy. (My words, of course, not his.)
• The initial Greenplum/EMC product vision appears truly centered around “private cloud,” specifically including Greenplum, VMware, and EMC storage arrays.
• Some other areas of potential Greenplum/EMC technical synergy I think are cool obviously haven’t been seriously addressed yet.
• Based on what I heard from Ben about the aura around the deal and also on what I know of the individual executives at Greenplum, I think each of them is a good bet to stick around EMC for a while. (That’s on average. Of course, it would be surprising if 100% of them stayed around very long.) Basically, there’s at least a chance EMC/Greenplum will do some pretty cool stuff, and most of the guys will probably stick around to see if that actually starts to happen.*

*Also, when they do eventually leave, they’ll surely say things to the effect “The cool stuff is well underway; my work here is done.” That party line is almost guaranteed, no matter how things unfold in reality.

The acquisition of Greenplum will be an all-cash transaction and is expected to be completed in the third quarter of 2010, subject to customary closing conditions and regulatory approvals. The acquisition is not expected to have a material impact to EMC GAAP and non-GAAP EPS for the full 2010 fiscal year. Upon close, Bill Cook will lead the new data computing product division and report to Pat Gelsinger. EMC will continue to offer Greenplum’s full product portfolio to customers and plans to deliver new EMC Proven reference architectures as well as an integrated hardware and software offering designed to improve performance and drive down implementation costs.

Greenplum is one of my biggest vendor clients, and EMC is just becoming one, but of course neither side gave me a heads-up before the deal happened, nor have I yet been briefed subsequently. With those disclaimers out of the way, some of my early thoughts include:

• I wish my clients would never buy each other, but it’s inevitable.
• I don’t think anybody evaluating Greenplum should be much influenced by this deal one way or the other. (Whether they will be is of course a different matter.)
  • EMC tends to run its bigger software acquisitions in a fairly hands-off manner. There’s no particular FUD (Fear/Uncertainty/Doubt) reason why this deal should stop anybody from buying Greenplum software.
  • I also don’t think adding a rich parent adds much of a reason to buy from Greenplum. But if you’re the type who’s nervous about smaller vendors — well, Greenplum now isn’t so small.
• Greenplum Chorus could, in principle, work with non-Greenplum DBMS. That possibility suddenly looks a lot more realistic.
• The list of analytic DBMS vendors with an appliance orientation is pretty impressive, including:
  • Oracle, with Exadata
  • Microsoft, partially
  • Teradata
  • Netezza
  • Now EMC/Greenplum, at least partially
  • Weaker players such as:
    • The ailing Kickfire, which a client (not Kickfire itself) tells me is being shopped around
    • The reeling HP Neoview
    • XtremeData, but I’m still waiting to hear of XtremeData’s first real sale
• Greenplum is something of a specialist in large databases. EMC has to love that.
• Greenplum’s weakness is concurrency.
• Greenplum’s “polymorphic storage” is a good fit for a storage vendor with appliance-y ideas.
• And finally — I think that even software-only analytic DBMS vendors should design their systems in an increasingly storage-aware manner, and have been advising my vendor clients of same. I’ll blog that line of reasoning separately when I get a chance, and edit in a link here after I do.

Related links (edit)

• Here’s the promised post as to why analytic DBMS need to be ever more storage-aware.
• Dave Kellogg crunched the EMC/Greenplum numbers, coming up with an estimated valuation range of $3-400 million, the high end of which is rumored to be correct.
• Merv Adrian suggests the big EMC/Greenplum loser is ParAccel, a viewpoint which presumably presupposes that the EMC/ParAccel partnership was significant in the first place.
• I talked with Ben Werther and posted more about Greenplum and EMC.

• Tom Foremski outlined the dire (at least in theory) privacy risks of geolocation services, going into a lot more detail on that point than I ever have. However, he topped that off with the odd claim that people pay toll (rather than using an electronic service) to cross the Bay Bridge because they fear being tracked, rather than for reasons of time or money.
• Oracle had an earnings conference call. Larry Dignan did a good job of covering the highlights; the gory details are on the Seeking Alpha transcript, especially pp. 3-5.  Oracle now claims to be getting lots of multi-system deals for Exadata. (But I still haven’t seen much in the way of production customers named.) ULAs, which I presume are Unlimited License Agreements, are important on the software side. Besides picking on IBM and SAP, Oracle even touted a competitive win vs. EMC, which not coincidentally seems to be working on partnering with almost every Oracle competitor it can find.
• Brian Prentice of Gartner basically accused open source of being Dotcom 2.0, in terms of dubious business models and the hype associated with same. I agree with many of his particulars, and indeed often steer vendor clients away from open source strategies. For marketing purposes, I do feel that sometimes free can be a real cool price; but open source is not the only way to be free.
• Akiban, which I wrote about a couple of days ago, seems to be building out its website. As of this writing the website is still pretty raw, with bewildering messaging, carelessly repeated paragraphs, and a notable lack of clues as to who’s in company leadership. Even so — unless I missed some of the current stuff before, the site has come a long way in a few days, so maybe there’s hope.
• Groovy Corporation, which introduced the Groovy SQL Switch just last summer, seems to be doing something different now. It’s merged into a company called uCirrus (where the u is really a mu), but uCirrus doesn’t have a meaningful website yet, whereas Groovy does. There’s stuff there about a “push data cloud,” stressing the importance of not being a DBMS, under the name Cortex, whatever that all means. Groovy seems to have an online gaming deal for Cortex with MySpace, or maybe Cortex is just the name of a specific Groovy/MySpace project.
• Mike Mooney offered a long rant on the problems with database (design) version control. He did concede that the most recent Microsoft Visual Studio might help, for those who are bought into (and can afford) the Microsoft stack. Frankly, I think that’s what views are for, updatable or otherwise. In many cases, they’ll let you build what you need, quickly and without breaking anything, and you can leave it to the DBAs to sort out database performance later.
• I just discovered Chad Stewart’s programming blog. While he’s evidently a game programmer, a lot of his comments have broader applicability.
• Chip Hazard offered a VC’s perspectives on the difficulties facing enterprise IT startups. (Hat tip to Miriam Tuerk for turning me on to him.) Although he didn’t phrase it this way, his bottom line (at least the part I agree with) is that the startup’s products have to be amazingly superior to the alternatives (big vendors or in-house).

There’s an interesting statistic floating around that something like 20% of all enterprise storage is used in DW/BI environments.  On that basis alone, it shouldn’t be a surprise that EMC would take a serious interest in this topic.

Hmm. It’s hard to think of anything in an OLTP database, log file, whatever that shouldn’t be copied into a data warehouses at this point.  Sometimes it should then be discarded, sometimes not.  E-mail, other text-based messaging, other media, etc. are an exception to that rule, however.

OK, I can believe the 20% figure.

Sure, there’s always more data and more users, but occasionally a data warehouse grows from simply being a nice-to-have decision support application into an operational part of the business — and the game changes.

True. On the other hand, older data warehouses are most commonly managed either by general-purpose DBMS such as Oracle and DB2, or else by Teradata, and generally designed for more robustness than perhaps they even needed.

But I guess Sybase IQ and Essbase are two big exceptions to that generalization.

… usually there’s no “one smokin’ query” to go consider — the reality is usually hundreds of ad-hoc requests, some optimized, some not, all hammering the data store.  Designing for optimal performance isn’t always about maximizing sequential access.  Unoptimized queries can hit the data store with random reads, radically changing the performance profile.

I.e., workloads aren’t trivially simple. But Chuck’s way of saying it has more dramatic flair. 

Many operational DW/BI environments are now continually updated to provide near-real-time results to queries, meaning write performance now becomes a very interesting topic.

Environments with very large numbers of spindles also may have to figure in rebuild times for failed disks (they do fail occasionally, you know …) and considering what the resulting performance impact might look like.

More and more DW/BI environments are being designed as HA environments — with varying degrees of redundancy and failover throughout — up to and including a remote failover site.

Good points all.

And, oh yes, we’ve got an entirely new magic ingredient to play with — enterprise flash drives.  While usually impractical for the primary data store, they can do amazing performance magic when applied against the temporary caches used by the DW/BI application.

DBMS of different architectures seem to differ as to the extent they write intermediate result sets out to disk.  But yes — that can be a very big deal.

As to whether that’s all flash is usually practical for in data warehousing — today that’s probably true.

Then there’s the operational aspect of all this — how is it all backed up and recovered?  Waiting days or weeks for a 100TB environment to be recreated from tape or source data isn’t usually an option for most businesses.

What about development, testing and staging of larger DW/BI applications?  Things like snaps and replication come into play.  And, yes, we’ve got more than a few customers doing remote DR for their DW/BI environments.

More good points.

Stepping back from the application itself, all DW/BI environments produce mountains of downstream data — lots of query results, analysis, data cubes, reports, and more.  Sometimes these downstream environments can be much larger than the DW/BI that creates them all.

That shouldn’t be happening.  If it is, you should get a fast analytic DBMS and reduce your dependence on OLAP cubes.

Over the last few years, I’ve been amazed at how many new software players are coming into this application space.

Sure, we’re working with industry standard players like Oracle, IBM and Microsoft.  We’re also starting to do more work with the second wave of appliance vendors like Teradata and Netezza.

Most interesting to me are the newest wave of software-only players who can take a relatively standard scale-out server/storage environment and do some amazing things: vendors like GreenPlum, Vertica, DatAllegro and ParAccel.

The market success of those four seems inversely proportional to the supposed closeness of their relationships with EMC.

Just teasing, Chuck.

The other heated debate in the industry is the “dedicated appliance vs. standardized infrastructure”.

Some will argue that a customized and bespoke all-in-one environment is best for DW/BI applications.  EMC’s point of view is that intelligent and optimized use of standardized infrastructure can deliver similar — or sometimes better — results, and deliver an operational environment that works pretty much the same as the rest of the landscape.

One of the useful aspects of our new Competency Center is that all of these approaches can be evaluated — side-by-side if needed — using a relatively standard server/storage infrastructure environment as a starting point.

Non-EMC alternatives can be evaluated side-by-side in EMC’s facilities with EMC-based ones??

Somehow, I don’t think that’s exactly what Chuck meant …

And then there’s the extremely interesting prospect of running DW/BI environments under VMware.  We’ve already established that there’s no disk I/O tax using VMware, and in some cases we get better I/O results.

Interesting!

More intriguing is the ability for VMware to take modern four-socket, six-core large-memory server designs, and partition them into comfortable virtualized chunks that DW/BI software can comfortably exploit, with the potential of delivering substantially more aggregate server performance in a given server environment.

As with the OLAP cube references above, this makes me think Chuck is thinking not just of high-end but also of sub-terabyte data warehousing.

Unlike DATAllegro, which manages all of your data on EMC boxes, or ParAccel, which wants to manage some of it on EMC, Netezza will continue to manage data solely on direct-attached disk, using CLARiiONS just as staging areas. The cutesy name for this is “Storage pad”, unless I misheard and it’s really “Storage pod”; the cutesy catchphrase is “TIVO for data management.” Highlights of what is to be staged include:

• Backups. And EMC backup software will be exploited.
• Replication. Netezza customers are increasingly requesting stronger capabilities in high availability and disaster recovery.
• Data loading.

Note: As per my new policy regarding embargoes, I scheduled this to appear Tuesday morning, May 20, at 8:45 am. It didn’t work, and I’m posting it manually some hours later. Back to the drawing board …

• ParAccel’s usual shared-nothing configuration is hooked up to SAN-based EMC storage at the back end.
• Around half of the total data is on internal (i.e., node-specific) disks, mirrored on the storage device. The rest of the data lives only on the EMC device. Logically, all this data is integrated. So hopefully you’ll be able to process more data per unit of time than you could on a standard ParAccel configuration.
• Also, different parts of the EMC device are dedicated to different ParAccel nodes. So, while this isn’t a shared-nothing architecture, at least it’s shared-not-very-much. (DATAllegro does something similar, although without the mirroring on direct-attached storage.)
• Backup, snapshotting, and so on are inherited from EMC. Administration will increasingly be integrated with EMC’s.

• Sun partners with both Greenplum and ParAccel.

• HP sells Neoview, and also is partnered with Vertica.

• EMC (together with Dell in North America and Bull in Europe) sells DATAllegro. Now EMC is also entering a partnership with ParAccel.

• Teradata is pretty far down the road toward releasing a low-end product.

In addition, multiple hardware vendors have “reference architecture” technical arrangements with Oracle, to try to capture some of the benefits of appliances. And IBM is constantly in partnership discussions with data warehouse specialists, notwithstanding having multiple data warehouse offerings of its own.

Positioning of these various offerings is confused. Part of the reason is the large vendors’ postures “We’re big and trustworthy, and those little upstart vendors aren’t – until the moment we partner with one of them.” Part of the reason is the small vendors’ stances of “We can do all things for all people – and by the way, 9 of the 14 customers we’ve ever had are all doing pretty much the same thing.” And part of the reason is just an industry penchant for secrecy.

To a first approximation, I think there are two sensible ways to define the tiers. In each case, we’re talking about what kinds of databases the various products are suited for.

• Criterion S (for “Size”). “Bigger than Oracle can handle” vs. “Small enough that Oracle can handle it” (but that depends on what the definition of “handle” is).

• Criterion U (for “Usage”). “Full enterprise data warehouse” vs. “big honking data mart”.

But those are very different classification rules – many products that might be upper-tier by Criterion S are lower-tier by Criterion U, and vice-versa. For example:

• Teradata’s current products are at the upper end by either criterion. Even so, a significant fraction of older Teradata installations are below 5 terabytes or even 1 terabyte in size.

• More generally, Teradata emphasizes Criterion U. Hence any future low-end products will surely be positioned as lower-tier by that criterion. Beyond that, I wouldn’t be surprised if release is delayed, with the final version of those products being different than what previously leaked. E.g., they might well be designed to compete with newer vendors that are upper-tier by Criterion S.

• Netezza has clearly made it into the upper tier by the Size criterion. Most of its installations are lower-tier by Criterion U, but it trumpets a few exceptions that it describes as “enterprise data warehouses” in success stories.

• DATAllegro is upper tier by Criterion S — more so than any other vendor except Teradata, in that there are at least two credible stories of DATAllegro warehouses at or above the quarter-petabyte mark. Even so, DATAllegro is still mainly in the lower tier by Criterion U. I.e., the most natural use of DATAllegro technology is to build Very Big data marts.

• Vertica is a purely lower-tier Criterion U player, given its focus on single fact table schemas. But it’s well on its way into the upper tier by Criterion S.

• Dataupia straddles the boundary of the tiers by Criterion S. That is, it’s meant to offload existing Oracle, SQL Server, or DB2 databases, or in some OEM cases to be a cheaper alternative. That sounds lower-tier. On the other hand, it has one 120 terabyte reference, which puts it squarely in in the upper tier. By Criterion U it’s pretty lower-tier.

• ParAccel seems lower-tier by either criterion. And I’m too burned out on ParAccel’s secrecy to probe hard for exceptions.

• Oracle, MS SQL Server, et al. are – pretty much by definition – lower-tier by Criterion S, but upper-tier by Criterion U.

• HP Neoview is obviously meant to get to the higher end by both criteria. But like most specialty products, right now it’s further along by the Size criterion than the Usage one. Even so, it seems no further along by Criterion S than partner HP’s partner Vertica is.

• Greenplum has clearly gotten to the upper tier by the Size criterion. But like most of the competition, it still seems to be in the lower tier by Usage.

• Infobright is in the lower tier by either criterion. (They don’t even have an MPP offering yet.)

• Kognitio KX2 is in the lower tier by either criterion. However, Kognitio aspires to move up when measured by Usage.

• The last time I looked, Sybase IQ was lower tier by either criterion.

Related links:

• A quick survey of data warehouse management technology
• Data warehouse appliances — fact and fiction

Please subscribe to our feed!

The talk is called Highly Scalable Analytic Appliance Powered by EMC and ParAccel, and the abstract says:

Large and medium size enterprises are struggling with the technical and business challenges associated with processing operational data in near real time while executing increasingly complex queries on multi-terabyte data warehouses. To address these challenges EMC and ParAccel have jointly engineered and developed a highly scalable and performant analytic appliance. This solution is built on EMC CLARiiON midrange CX-3 UltraScale networked storage and ParAccel’s analytic columnar data store. Customers can simply deploy the EMC/ParAccel analytic appliance by simply extending their existing EMC footprint on enterprise ready storage while leveraging EMC’s proven solutions.

Hat tip to Mark Madsen.

*How’s that for a nice, vague euphemism?

DATAllegro is targeted at warehouses sized, at a minimum, in the tens of terabytes of user data. Oracle’s technology works well enough up into at least the multi-terabyte range — unless you’re looking to get the best possible price and/or performance on your system — but then things start getting dicey. So there isn’t a lot of overlap between the two Dell/EMC offerings.

And if I don’t stop right there, I’m going to start venting about NDAs, especially when I’m told “Oh, this isn’t for our sake, but rather because our big-company partner(s) demands it.” So stay tuned.

I just recently encountered some old notes in which Sybase proudly announced a “1 gigabyte challenge.” The idea was that 1 gig was a breakthrough size for business databases.

Time flies.