*I probably have it worse than most people in that regard, because my clients run tentative feature lists and announcement schedules by me well in advance, which may get changed multiple times before the final dates roll around. I also occasionally miss some detail, if it wasn’t in a pre-briefing but gets added at the end.

Anyhow, the three big themes of this month’s announcements are probably:

• Integrating different kinds of analytic processing into databases and DBMS.
• Taking advantage of hardware advances.
• Playing catchup in areas where small vendors’ products weren’t mature yet.

For example, the three biggest data warehouse DBMS product announcements this month are probably:

• Aster Data nCluster 4.5. Much like Aster’s prior release — Aster Data nCluster 4.0 – Aster Data nCluster 4.5 has a major focus on integrating analytics and database processing. This time, the emphasis is on application development tools and pre-built analytic packages. In addition, Aster’s management tool GUIs have been upgraded, building on catch-up functionality in the Aster Data nCluster 4.0.
• Netezza’s “i” add-on to its existing TwinFin products. With Netezza TwinFin(i), Netezza becomes the second MPP RDBMS vendor with a comprehensive “Big Data Analytic Platform” kind of strategy. (Netezza would surely argue that it was the first, but that depends on how seriously one took Netezza’s prior attempt.) Many of the details are different from Aster’s, of course, but the general philosophy is similar. So far, Netezza has announced one interesting proprietary library of analytic packages (for linear/matrix algebra), plus the port of 4,000 or so functions in open source libraries.
• Vertica 4.0. Vertica has had a highly innovative columnar DBMS architecture from the getgo, but at the cost of some restrictions or awkwardness in the relationship between data layout and SQL processing. Vertica says that Vertica 4.0 fixes all that. In addition, it has some analytic processing enhancements, especially in the time series area, where Vertica doesn’t vigorously dispute that Sybase IQ previously had an advantage.

In addition,

• Teradata is announcing its Data Warehouse Appliance 2580, the successor to the Teradata 2550. This is purely a hardware refresh; Teradata’s hardware and software upgrades are not generally synced. The Teradata 2580 upgrades CPUs from Harpertown to Nehalem, includes 3X the RAM of its predecessor, and offers an option for 1 TB disks (thus lowering the bottom price/TB a lot, to $31K list).
• Aster, Vertica, and ParAccel have all called attention to the fact that, if solid-state drives have interfaces like those of disk drives, and if a DBMS supports disk drives, then a DBMS also supports solid-state drives as well. At least Aster and ParAccel have signaled that they have at least one customer or prospect each interested in Fusion I/O’s solid-state technology, especially in the retail sector. This is basically a hardware matter as well, and a big deal only for those who were somehow unaware of the impending dominance of solid-state memory technology.
• Sybase announced its Aleri acquisition earlier this month.
• Various vendors have bragged about various rankings, awards, or benchmarks, or – sometimes less tediously — about last year’s sales results.

• Netezza’s software runs on well-designed proprietary hardware. Aster runs on hardware that’s more off-the-shelf.
• Aster was first to ship, and will also be first to ship an IDE (Integrated Development Environment).
• MapReduce is central to Aster’s approach. Netezza TwinFin(i) supports MapReduce too, specifically a Hadoop implementation, but I don’t get the sense that everything Netezza does is built on MapReduce underpinnings.
• Both Aster and Netezza try to provide rich functionality for creating in-memory data structures parallel analytic programs can use. Both seem to let you escape from the pure relational-table paradigm more easily than, say, Teradata’s new persistent memory capabilities do.
• Aster and Netezza have made different choices about what kinds of prebuilt analytic packages to offer. Netezza could actually leapfrog Aster in this regard, but let’s see where each vendor is by, say, mid-year. If you care about the details of built-in analytic functions, you really should consider executing non-disclosure agreements with both those companies.
• Both Aster and Netezza stress that you can run analytic functions out-of-process, greatly reducing the chance that they crash the database. Netezza and I’m pretty sure also Aster also retain the option of running in-process, which provides maximum performance. (In Netezza’s case C++ is the only in-process language supported, and I think Aster has a similar limitation.)
• Like Aster, Netezza is integrating SQL queries and other analytic processing under the same workload management rubric.
• Much like Aster, Netezza is tap-dancing by implying much richer forthcoming SAS support than anything currently announced. (The crunch-per-paragraph ratio in either vendor’s SAS-related press releases to date is distressingly low.)

More specifically, here are some highlights of what I know, am guessing, and/or am allowed to say about Netezza TwinFin(i) at this time.

• The foundation for the analytic add-ons in Netezza TwinFin(i) is some sort of low-level “analytic executables.” Not understanding exactly what these are is my biggest area of confusion in the whole TwinFin(i) stack. Are they all C++, with everything translated into same? Is there Java all the way down as an alternative? (E.g., Hadoop is written in Java.) Anyhow, whatever it is, it’s surely a big improvement on Netezza’s prior Verilog-based generation of analytic extensibility technology.
• The announced list of languages supported in Netezza TwinFin(i) is Java, Python, Fortran, R, and C/C++. More are coming.
• Netezza has named a lot of analytic functions it is adding, and hinting about more to come. It has named CRAN/R and GNU libraries, saying those have 1900 or more functions each. Netezza has also built its own linear algebra library for TwinFin(i), called nzMatrix. And as previously noted, TwinFin(i) also boasts a Hadoop implementation.
• I haven’t heard about much in the way of TwinFin(i)-specific IDE support.
• I don’t really have details as to what kinds of in-memory data structures Netezza TwinFin(i) does or doesn’t support.

• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

*Links to Gartner Magic Quadrants commonly break, but that one worked at the time of this posting.

• Gartner thinks that data warehouse appliances are on the rise, due to their simplicity.
• Gartner correctly says that Teradata has been a data warehouse appliance vendor from the getgo.
• Gartner characterizes IBM as being an appliance vendor as well.
• Gartner suggests that HP is having trouble living up to its technical promises for Neoview.
• Gartner further suggests — no surprise here — that HP Neoview has had very few new customers past its initial wave.
• Gartner notes IBM’s difficulties in selling data warehouse installations of DB2, despite what on paper is great-sounding technology.
• Gartner says — also no surprise — that illuminate “has seen little success in North America since opening its first office in the U.S. over two years ago.”
• Ingres has evidently gotten a few BI-centric “appliance” deals, e.g. with Jaspersoft. But basically Ingres isn’t doing well in data warehousing.
• Gartner does say Ingres has “the strongest open-source DBMS offering for data warehousing.” Being very literal about “open source,” that’s a defensible claim — but it’s pretty irrelevant in a world where Greenplum Single-Node Edition can be had for free. It also waves away all the data mart use cases in which Infobright Community Edition shines.
• Gartner says that Netezza is working out as a “complex workload” enterprise data warehouse provider, according to reference checks, in addition to its established success in data mart scenarios.
• Gartner says Oracle’s offering has finally become “accepted” in the market for databases >50 TB. I guess I can live with that fairly weak claim, but I wouldn’t go much further than that.
• Gartner asserts that, unlike software-only Oracle, Oracle Exadata isn’t significantly harder to administer than “other mixed OLTP/OLAP DBMS vendors,” because Exadata is fast enough you don’t need to jump through all those hoops any more to get tolerable performance. The money quote is “one reference reported reducing the number of indexes by a factor of 100 to fewer than five.” Note, however, that Gartner does not seem to assert that Exadata’s ease of use rivals that of the newer analytic DBMS specialists.
• Gartner confirms Oracle’s reluctance to do onsite Exadata POCs, but says it is not absolute. This is roughly compatible with what I’m hearing elsewhere, and indeed with Oracle own claims to be ramping up availability of Exadata POC hardware.
• Gartner’s criteria for inclusion include at least 10 different organizations having a product “in production.” Thus, the big surprise was ParAccel being included. The money quote there is “With approximately 20 customers in the pharmaceutical, retail, financial and media/advertising analytics sectors, ParAccel has a good reference base.” That assessment is difficult to reconcile with other information, but I’ve been told Gartner is sticking to its guns. That assessment would be even harder to believe if those 20 references were all alleged to be true production customers.
• Gartner notes that you basically can’t run a 1 TB+ MySQL data warehouse without sharding. (Of course, Infobright has an alternative, and up to a small number of terabytes so does Kickfire.)
• Gartner reports that at least some customers are pleased with Sybase IQ’s mixed workload/enterprise data warehouse capabilities.
• Gartner correctly notes that Oracle Exadata is a price-competition challenge for Teradata.
• Gartner notes that 20% of Vertica’s customers are outside the US. While not shocking, that’s more than I realized.
• Gartner notes something I don’t think I’ve posted yet, which is that Vertica has a customer with 300 TB of data. (The identity is a deep dark secret, but if I told you you probably wouldn’t recognize the name anyway.)

As does any such piece, the Gartner Data Warehouse DBMS Magic Quadrant also has outright errors.  For example:

• Aster Data isn’t really “the newest entrant to the DBMS data warehouse world.”
• Aster’s SQL/MapReduce was not new in Release 4.0.
• Greenplum isn’t yet pushing down code to the storage tier.
• I’m not sure what kind of database-tier parallelism Gartner is claiming is new in Oracle in 11g Release 2 — but I doubt it’s really new. Rather, what Oracle has done recently is make parallelism less administratively cumbersome.
• Vertica wasn’t really the first DBMS in the cloud. At most it was the first pure-play analytic DBMS to get there.

That said, highlights of my Netezza Skimmer briefing included:

• In essence, Netezza Skimmer is 1/3 of Netezza’s previously smallest appliance, for 1/3 the price.
• I.e., Netezza Skimmer has 1 S-blade and 9 disks, vs. 3 S-blades and 24 disks on the Netezza TwinFin 3.
• With 1 disk reserved as a hot spare, that boils down to a 1:1:1 ratio among CPU cores, FPGA cores, and 1-terabyte disks on Netezza skimmer. The same could pretty much be said of Netezza TwinFin, the occasional hot-spare disk notwithstanding.
• Netezza Skimmer costs $125K.
• With 2.8 or so TB of space for user data before compression, that’s right in line with the Netezza price point of slightly <$20K/terabyte of user data.
• That assumes Netezza’s usual 2.25X compression. I forgot to ask when 4X compression was actually being shipped.
• I forgot to ask, but it seems obvious that Netezza Skimmer uses identical or substantially similar components to Netezza TwinFin’s.
• Netezza Skimmer is 7 rack units high.
• In place of the SMP hosts on TwinFin Systems, Netezza Skimmer has a host blade.
• Netezza (specifically Phil Francisco) mentioned that when Kalido uses Netezza Skimmer for its appliance, there will be an additional host computer, but when it uses TwinFin for the same software, the built-in host will suffice. (Even so, I suspect it might be too strong to say that Skimmer’s built-in host computer is underpowered.)
• Netezza also suggested that more appliance OEMs are coming down the pike specifically focused on the affordable Skimmer.

Obviously, Netezza Skimmer isn’t breaking any new technical ground. If Netezza had just called Skimmer “TwinFin 1,” nobody should have objected. So the main news here is that you can buy a Netezza box for $125K, plug it in, load a few terabytes of data, and be good to go with a pretty solid data warehouse.  For enterprises and data mart outsourcers with databases of the appropriate size, that could be a pretty attractive deal.

Is Netezza Skimmer as cheap as buying your own hardware and putting (free) Greenplum Single-Node Edition software on it? Not even close, especially since Greenplum’s free option limits you to lower overall compute power. Does Netezza Skimmer have as high availability as more expensive alternatives? In some cases, surely not. Skimmer is neither the cheapest thing around nor an utterly high-end product.

But Netezza Skimmer belongs on a lot of short lists even so.

• Teradata ascribes considerable importance to a Virtual Storage technology whose main purpose is to allow mixing of heterogeneous storage devices in a single system. And the discussion rarely suggests that these parts will be in a rigid fixed relationship.
• Netezza — as Teradata keeps reminding me — often sells boxes with the expectation that they won’t be filled with data, so as to increase spindle count and hence performance.
• Oracle/Sun have dropped some comments about Exadata being more flexibly configured going forward.
• Kickfire’s new “high-end” appliance lets you attach fairly arbitrary amounts of external storage.
• And of course, software-only analytic DBMS vendors run their software in all sorts of hardware and storage environments.

What’s more, the claim never made a lot of sense anyway. With the rarest of exceptions, even a single data warehouse’s workload will contain different queries that strain different parts of the system in different ratios. Calculating the “ideal” hardware configuration for that single workload would be forbiddingly difficult. And even if one could calculate it, it almost surely would be different than another user’s “ideal” configuration. How a single hardware configuration can be “ideally balanced” for a broad class of use cases boggles the imagination.

]]> http://www.dbms2.com/2009/10/25/data-warehouse-balanced-hardware-configuration/feed/ 6 http://www.dbms2.com/2009/10/05/oracle-exadata-2-capacity-pricing/ http://www.dbms2.com/2009/10/05/oracle-exadata-2-capacity-pricing/#comments Mon, 05 Oct 2009 12:20:19 +0000 Curt Monash http://www.dbms2.com/?p=1021 Summary of Oracle Exadata 2 capacity pricing

Analyzing Oracle Exadata pricing is always harder than one would first think. But I’ve finally gotten around to doing an Oracle Exadata 2 pricing spreadsheet. The main takeaways are:

• If we believe Oracle’s claims of 10X compression, Exadata 2 costs more per terabyte of user data than Netezza TwinFin — $22-26K/TB vs. TwinFin’s <$20K — but less than the Teradata 2550.
• These figures are highly sensitive to assumptions about Oracle’s hybrid columnar compression.
• Similarly, if Netezza or Teradata were to significantly upgrade their own compression, the price comparison would look quite different.
• Options such as Data Mining or Oracle Spatial add 12% or so each to Exadata’s total system price.

Longer version

When Oracle introduced Exadata last year it was, well, expensive. Exadata 2 has now been announced, and it is significantly cheaper than Exadata 1 per terabyte of user data, based on:

• Similar overall pricing
• Twice the disk capacity
• Better compression

Compression is the big question mark. Row-based DBMS vendors have traditionally been, if anything, conservative in their compression claims, although Netezza recently went with a not-sandbagged 2.25X compression estimate to get below the $20K/terabyte price point. Columnar software vendors have tended to be more aggressive, with figures of 10X or more casually thrown around, or 40-50X for archival storage. But since columnar vendors sell mainly on a software-only basis, those claims haven’t generally shown up in per-terabyte total system cost comparisons, which most commonly focus on data warehouse appliance product lines.*

*Kickfire, the one columnar pure-play appliance vendor, hast to date been quite conservative in its compression marketing claims.

Oracle, however, recently announced a feature called hybrid columnar compression, and is now making compression claims with the usual columnar grandiosity. Oracle’s story is 10X compression, and they’re sticking to it, perhaps because 10 is such a nice round number.* Since Oracle hybrid columnar compression is part of 11g Release 2, and isn’t Exadata-specific, wWe can hope to eventually get a sense from the field of what levels of compression are actually realistic. (Edit: Actually, it seems that hybrid columnar compression only works with Exadata, at least at this time.) But for now, we don’t have much to go on except Oracle’s claims.

*Greg Rahn of Oracle tweeted me yesterday that one customer is getting 12-17X compression on “dimensional model” data. That sounds comparable to Vertica’s claim of 20X on “marketing analytics” and 30X on “consumer data” datasets.

Based on 10X compression (vs. Netezza’s 2.25X and Teradata’s lower figure), Oracle Exadata 2 is somewhat more expensive than Netezza TwinFin, and significantly cheaper than Teradata’s 2550. Specifically, Oracle Exadata 2 comes in around $22K/TB of user data for a full rack, and $26K/TB for a quarter rack, which is the Exadata 2 configuration more comparable to a TwinFin rack in user data capacity. This is if you look at the Exadata hardware version that uses 600 GB SAS drives (vs. 1 TB SAS drives for TwinFin and 300 GB SAS drives for Teradata). With 2 TB SATA drives, at the same system pricing, Exadata prices are 70% lower, getting down to $6K/TB for a full rack. You can see how I got these figures on my Oracle Exadata 2 pricing spreadsheet linked above.

Obviously, price/terabyte is just one metric. Throughput is often even more important, but also is a lot harder to quantify simply. Oracle Exadata 2 offers more raw I/O than Netezza TwinFin. Netezza TwinFin, with its FPGA-based pipelining, probably has more processing oomph than Oracle Exadata. Oracle’s compression could lead to better use of RAM cache. And so it goes.

Meanwhile, two factors that in my opinion don’t matter much to the analysis are:

• The re-usability of Oracle licenses on other hardware. Most of Exadata’s cost is either for hardware, or for server software that’s priced on a per-core basis. Neither of those is going to manage much (or any) more data three years from now than it can today.
• What Oracle claims as pricing metrics. Oracle’s comments on Exadata pricing generally sow confusion, which is why I do my own spreadsheets.

Related links

• The hunt for Oracle Exadata production references (but hopefully some will be revealed at Oracle Open World)
• A favorable rumor about Exadata sales
• Issues in integrating OLTP and data warehousing in a single system
• James Kobielus of Forrester tweets that there are “plenty” of pleased Exadata users
• More on Oracle Exadata hybrid columnar compression

]]> http://www.dbms2.com/2009/10/05/oracle-exadata-2-capacity-pricing/feed/ 13 http://www.dbms2.com/2009/09/30/facts-and-rumors/ http://www.dbms2.com/2009/09/30/facts-and-rumors/#comments Wed, 30 Sep 2009 06:21:44 +0000 Curt Monash http://www.dbms2.com/?p=963

]]> http://www.dbms2.com/2009/09/30/facts-and-rumors/feed/ 2 http://www.dbms2.com/2009/09/29/a-c-nielsen-data-warehousing-dbms/ http://www.dbms2.com/2009/09/29/a-c-nielsen-data-warehousing-dbms/#comments Tue, 29 Sep 2009 15:04:26 +0000 Curt Monash http://www.dbms2.com/?p=950 In its latest earnings call, Oracle made a reference to The Nielsen Company that was — to put it politely — rather confusing. I just plopped down in a chair next to Greg Goff, who evidently runs data warehousing at Nielsen, and had a quick chat. Here’s the real story.

• The Nielsen Company has over half a petabyte of data on Netezza in the US. This installation is growing.
• The Nielsen Company indeed has 45 terabytes or whatever of data on Oracle in its European (Customer) Information Factory. This is not particularly growing. Nielsen’s Oracle data warehouse has been built up over the past 9 years. It’s not new. It’s certainly not on Exadata, nor planned to move to Exadata.
• These are not single-instance databases. Nielsen’s biggest single Netezza database is 20 terabytes or so of user data, and its biggest single Oracle database is 10 terabytes or so.
• Much (most?) of the rest of the installations are customer data marts and the like, based in each case on the “big” central database. (That’s actually a classic data mart use case.) Greg said that Netezza’s capabilities to spin out those databases seemed pretty good.
• That 10 terabyte Oracle data warehouse instance requires a lot of partitioning effort and so on in the usual way.
• Nielsen has no immediate plans to replace Oracle with Netezza.
• Nielsen actually has 800 terabytes or so of Netezza equipment. Some of that is kept more lightly loaded, for performance.

AC Neilsen, for instance, we deployed a 45-terabyte data [mart], they called it; Adidas, 13 terabytes; Australian Bureau of Statistics, 250 terabytes; and of course, some of our high-end ones that you have probably heard of in the past, AT&T, 250 terabytes; Yahoo!, 700 terabytes — just gives you an idea of the size of the databases that are out there and how they are growing, and that’s driving the need for greater throughput.

I don’t know what’s being counted there, but I wouldn’t be surprised if those were legit user-data figures.

Some other notes:

• The Yahoo database is of course Yahoo’s first-generation data warehouse, which has been largely superseded by an internal system more than 10X that size. (Edit: Actually, Greg Rahn of Oracle says below that it’s a different database.)
• I’m keynoting the Netezza road show this month, and Nielsen is up there on stage touting Netezza. (Edit: Nielsen indeed does the overwhelming majority of its data warehousing on Netezza.)
• I’d be surprised if AT&T’s largest data warehouse were “only” 250 terabytes in size. (Edit: Actually, I am told the database in question is 310 TB of user data and growing. More later, hopefully.)
• Oracle didn’t exactly say that those were Exadata installations.