Data warehouse storage options — cheap, expensive, or solid-state disk drives
This is a long post, so I’m going to recap the highlights up front. In the opinion of somebody I have high regard for, namely Carson Schmidt of Teradata:
- There’s currently a huge — one order of magnitude — performance difference between cheap and expensive disks for data warehousing workloads.
- New disk generations coming soon will have best-of-both-worlds aspects, combining high-end performance with lower-end cost and power consumption.
- Solid-state drives will likely add one or two orders of magnitude to performance a few years down the road. Echoing the most famous logjam in VC history — namely the 60+ hard disk companies that got venture funding in the 1980s — 20+ companies are vying to cash in.
In other news, Carson likes 10 Gigabit Ethernet, dislikes Infiniband, and is “ecstatic” about Intel’s Nehalem, which will be the basis for Teradata’s next generation of servers.
| Categories: Data warehouse appliances, Data warehousing, eBay, Solid-state memory, Storage, Teradata | 16 Comments |
The SAP/Teradata deal explained
When I first saw the press release about the latest SAP/Teradata deal, I thought it sounded very Barney. But it turns out there’s a little bit of substance, as well. Amazingly, SAP BW doesn’t really run on Teradata right now. This deal will fix that. The time frame seems to be that SAP-BW-on-Teradata will ship with SAP BW 7.2 whenever that goes out. (First half of 2010?) Early adopters may be able to get their hands on it as early as Q3 2009.
Note: It surely would be more precise to insert “NetWeaver” a few times into that paragraph.
Just to be clear — I still don’t see this as a big deal. It doesn’t portend any grand SAP/Teradata joint mission to smite Oracle, IBM, and/or Microsoft. Nor is it a telling first step toward an SAP/Teradata merger. It just removes a particular competitive disadvantage Teradata had vs. Oracle et al., from which Teradata’s smaller specialist competitors still suffer. And it offers SAP BW customers another high-quality DBMS option.
| Categories: Business intelligence, Data warehousing, SAP AG, Teradata | Leave a Comment |
Vertica pricing and customer metrics
Since last fall, Vertica’s stated pricing has been “$100K per terabyte of user data.” Vertica hastens to point out that unlike, for example, appliance vendors or Sybase, it only charges for deployment licenses; development and test are free (although of course you have to Bring Your Own hardware). Offer the past few weeks, I’ve gotten other pricing comments from Vertica to the effect that:
- Of course, Vertica offers substantial negotiated quantity discounts. (Specifics that Vertica told me are confidential.)
- Actually,Vertica’s official price list (unpublished but apparently freely available to prospects) contains quantity discounts too.
- Finally, Vertica told me that its actual average price is around $25K/terabyte, and gave me person to publish same.
I didn’t press my luck and ask exactly what “average” means in this context.
As for customers, metrics I got include: Read more
Some DB2 highlights
I chatted with IBM Thursday, about recent and imminent releases of DB2 (9.5 through 9.7). Highlights included:
- DB2 is getting Oracle emulation, which I posted about separately.
- IBM says that it had >50 new DB2 data warehouse customers last year. I neglected to ask how many of these had been general-purpose DB2 customers all along.
- By “data warehouse customer” I mean a user for InfoSphere Warehouse, which previously was called DB2’s DPF (Data Partitioning Feature). Apparently, this includes both logical and physical partitioning. E.g., DB2 isn’t shared-nothing without this feature.
- IBM is proud of DB2’s compression, which it claims commonly reaches 70-80%. It calls this “industry-leading” in comparison to Oracle, SQL Server, and other general-purpose relational DBMS.
- DB2 compression’s overall effect on performance stems from a trade-off between I/O (lessened) and CPU burden (increased). For OLTP workloads, this is about a wash. For data warehousing workloads, IBM says 20% performance improvement from compression is average.
- DB2 now has its version of one of my favorite Oracle security features, called Label Based Access Control. A label-control feature can make it much easier to secure data on a row-by-row, value-by-value basis. The obvious big user is national intelligence, followed by financial services. IBM says the health care industry also has interest in LBAC.
- Also in the security area, IBM reworked DB2’s audit feature for 9.5
- I think what I heard in our discussion of DB2 virtualization is:
- Increasingly, IBM is seeing production use of VMware, rather than just test/development.
- IBM believes it is a much closer partner to VMware than Oracle or Microsoft is, because it’s not pushing its own competing technology.
- Generally, virtualization is more important for OLTP workloads than data warehousing ones, because OLTP apps commonly only need part of the resources of a node while data warehousing often wants the whole node.
- AIX data warehousing is an exception. I think this is because AIX equates to big SMP boxes, and virtualization lets you spread out the data warehousing processing across more nodes, with the usual parallel I/O benefits.
- When IBM talks of new autonomic/self-tuning features in DB2, they’re used mainly for databases under 1 terabyte in size. Indeed, the self-tuning feature set doesn’t work with InfoSphere Warehouse.
- Even with the self-tuning feature it sounds as if you need at least a couple of DBA hours per instance per week, on average.
- DB2 on Linux/Unix/Windows has introduced some enhanced workload management features analogous to those long found in mainframe DB2. For example, resource allocation rules can be scheduled by time. (The point of workload management is to allocate resources such as CPU or I/O among the simultaneous queries or other tasks that contend for them.) Workload management rules can have thresholds for amounts of resources consumed, after which the priority for a task can go up (“Get it over with!”) or down (“Stop hogging my system!”).
| Categories: Application areas, Data warehousing, Database compression, IBM and DB2, Market share and customer counts, OLTP, Parallelization, Workload management | 2 Comments |
IBM’s Oracle emulation strategy reconsidered
I’ve now had a chance to talk with IBM about its recently-announced Oracle emulation strategy for DB2. (This is for DB2 9.7, which I gather has been quasi-announced in April, will be re-announced in May, and will be re-re-announced as being in general availability in June.)
Key points include:
- This really is more like Oracle emulation than it is transparency, a term I carelessly used before.
- IBM’s Oracle emulation effort is focused on two technological goals:
- Making it easy for an Oracle application to be ported to DB2.
- Making it easy for an Oracle developer to develop for DB2.
- The initial target market for DB2’s Oracle emulation is ISVs (Independent Software Vendors) much more than it is enterprises. IBM suggested there were a couple hundred early adopters, and those are primarily in the ISV area.
Because of Oracle’s market share, many ISVs focus on Oracle as the underlying database management system for their applications, whether or not they actually resell it along with their own software. IBM proposed three reasons why such ISVs might want to support DB2: Read more
Clearing some of my buffer
I have a large number of posts still in backlog. For starters, there are ones based on recent visits with Aster, Greenplum, Sybase, Vertica, and a Very Large User. I suspect I’ll write more soon on Oracle as well. Plus there’s my whole future-of-online-media area. And quite a bit more will grow out of planned research.
So there are a whole lot of other worthy subjects I doubt I’ll be getting to any time soon. In some cases, of course, other people are doing great jobs of writing about same. Here are pointers to a few links that I am glad to recommend:
- I wrote recently that I’ve discovered a number of different in-memory OLAP engines. Cindi Howson far outdid that, writing at length for Intelligent Enterprise on in-memory analytics, in an article that seems to itself be a teaser for a longer, free white paper on the subject.
- CouchDB posted an eye-catching, risque slide presentation promoting CouchDB and, more generally, key-value stores, at least for internet applications. And yes, they’ve integrated MapReduce.
- Merv Adrian posted favorably about Birst, with special reference to its OEM efforts. As previously noted, I was highly unimpressed with Birst’s end-user BI story at the time of its September roll-out, and Jerome Pineau’s recent examination did nothing to reassure me. But perhaps OEM is a different matter.
- Merv also offers an interesting post about data integration upstart Expressor, and a highly favorable one about “visualization” vendor Tableau.
- Ann All interviewed Nigel Pendse, who grumped that BI features are overrated, and what end users really want is great query performance. I’m not so sure about the features side of that, but I’m hugely in agreement about the performance. That’s a big part of why the analytic DBMS industry is so vibrant. It’s also why in-memory OLAP is suddenly so hot.
DBMS transparency layers never seem to sell well
A DBMS transparency layer, roughly speaking, is software that makes things that are written for one brand of database management system run unaltered on another.* These never seem to sell well. ANTs has failed in a couple of product strategies. EnterpriseDB’s Oracle compatibility only seems to have netted it a few sales, and only a small fraction of its total business. ParAccel’s and Dataupia’s transparency strategies have produced even less.
*The looseness in that definition highlights a key reason these technologies don’t sell well — it’s hard to be sure that what you’re buying will do a good job of running your particular apps.
This subject comes to mind for two reasons. One is that IBM seems to have licensed EnterpriseDB’s Oracle transparency layer for DB2. The other is that a natural upgrade path from MySQL to Oracle might be a MySQL transparency layer on top of an Oracle base.
| Categories: ANTs Software, Dataupia, Emulation, transparency, portability, EnterpriseDB and Postgres Plus, IBM and DB2, Market share and customer counts, MySQL, Oracle, ParAccel | 11 Comments |
MySQL miscellany
For a guy who doesn’t go to the MySQL conference and routinely gets flamed by the MySQL community for being insufficiently adoring of their beloved product, I sure have been putting up a lot of MySQL-related posts recently. Here’s another, zooming through a few different topics. Read more
| Categories: MySQL, Open source | 4 Comments |
I don’t see why the GPL would be a major barrier to a useful MySQL fork
I posted suggesting that substantial elements of the MySQL community should throw their weight behind MySQL forks. Mike Olson of Cloudera helpfully pointed out, on Twitter and by email, how the GPL could appear to stand in the way of such an effort. But would it really?
Currently, any version of the MySQL code that isn’t proprietary to the MySQL company — which is owned by Sun and hence expected to be owned soon by Oracle — is covered by GPL 2. That license states (emphasis mine):
Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted,
Hence it is hard for me to see how the MySQL company could in any way hinder another software vendor from saying “Please buy my software, then go download a free copy of GPLed MySQL and run the two together.”*
| Categories: MySQL, Open source | 14 Comments |
This week is a REALLY good time to actively strengthen the MySQL forkers
As my first three posts on the Oracle/Sun merger suggested, I think Oracle will do a better job with MySQL product development than Sun has. But of course that’s a low hurdle. And so it leaves open the questions:
What should and/or will be the most widely adopted code lines of MySQL (or other open source DBMS),
especially for the types of users and vendors who are engaged with MySQL (as opposed to principal alternative PostgreSQL) today?
As much as I’ve bashed MySQL/MyISAM and MySQL/InnoDB for being low-quality general-purpose DBMS, I’d still hate to see MySQL-based development stall out. There are a number of MySQL engine providers with rather unique technology, that deserve a good front-end partner to build their products with. The high-volume sharding guys deserve the chance to continue down their current path as well. And so does the low-end mass market — although I’m least worried about them, as I can’t imagine any realistic scenario in which Oracle doesn’t offer a version of MySQL fully suited to support 10s of millions of WordPress and Joomla installations.
So far as I can tell, there are only four real and currently active candidates for MySQL code coordinator:
- MySQL itself, soon to be owned by Oracle.
- MariaDB, Monty Widenius’ proposed mainstream MySQL alternative
- Percona, which seems to have some fans as a superior alternative to vendor-supplied MySQL/InnoDB
- Drizzle, which is directly focused at web-centric MySQL users who never wanted a robust DBMS in the first place.
Patrick Galbraith and Steven Vaughan-Nichols did good jobs of illustrating the turmoil.
Oracle isn’t a very comfortable partner long term for the storage engine vendors, and Drizzle doesn’t seem to be what they need. So I think that Infobright, Kickfire, Tokutek, Calpont, et al. need to get aligned in a hurry with an outside MySQL provider such as Percona or MariaDB or a newcomer, preferably all with the same one. Yes, I understand that Infobright is getting a lot of marketing help from Sun these days, that Kickfire just got a nice-sounding Sun marketing announcement as well, and so on. But the time to start working toward the inevitable future is now.
And by “now” I mean “right now,” since the MySQL community is at this moment gathered together for its annual conference.
| Categories: Infobright, Kickfire, MySQL, Open source | 12 Comments |