DBMS2 -- DataBase Management System Services

Interpreting the results of data warehouse proofs-of-concept (POCs)

Curt Monash — Thu, 20 Nov 2008 01:08:35 +0000

When enterprises buy new brands of analytic DBMS, they almost always run proofs-of-concept (POCs) in the form of private benchmarks. The results are generally confidential, but that doesn’t keep a few stats from occasionally leaking out. As I noted recently, those leaks are problematic on multiple levels. For one thing, even if the results are to be taken as accurate and basically not-misleading, the way vendors describe them leaves a lot to be desired.

Here’s a concrete example to illustrate the point. One of my vendor clients sent over the stats from a recent POC, in which its data warehousing product was compared against a name-brand incumbent. 16 reports were run. The new product beat the old 16 out of 16 times. The lowest margin was a 1.8X speed-up, while the best was a whopping 335.5X.

My client helpfully took the “simple average” — i.e. the mean – of the 16 factors, and described this as an average 62X drubbing. But is that really fair? The median speed-up was only 17X. And in a figure I find more meaningful than either of those, the total reduction in execution time – assuming each of the reports was run the same number of times – was “just” 12 times.

Now, 12X is a whopping speed-up, and this was a very successful POC for the challenger. But calling it 62X is just silly, and that was the point of my earlier post.

So how should POC numbers be weighted? Ideally, one could calculate a big weighted sum: “Our daily workload will be a lot like 2,000 copies each of Queries 1, 2, 3, 4, and 5; 300 copies each of BigQueries 6 and 7; 25 copies of MegaQuery 7; and a copy of DestructoQuery 8; all multiplied by a factor of 17.”

But to come up with reasonable projections, it is not enough to look at past usage. After all, if the price of Query 3 goes down by 5X, while the cost of Query 8 goes down by a factor of 50, the relative consumption of Queries 3 and 8 is apt to change significantly. That’s just the economics of supply and demand.

Bottom line: The more accurately you can predict future data warehouse use, the more confidently you can choose the analytic database technology that’s best for you.

MySQL Query Analyzer

Curt Monash — Wed, 19 Nov 2008 23:21:28 +0000

Given how the product’s rollout has been handled, it seems necessary to comment on MySQL’s recently released MySQL Query Analyzer without actually having much information on the subject. Mark Callaghan offers a good take — he’s generally very favorable, but notes that MySQL has some limitations that Query Analyzer has trouble getting around.

Silly website tricks

Curt Monash — Tue, 18 Nov 2008 17:48:57 +0000

Vertica’s marketing is usually good-to-outstanding, but they made a funny misstep this time. If you go to the Vertica home page, you’ll see seasonal art suggesting that their product is a turkey and/or that it’s terrified it’s about to get the ax.

Live by the pun, die by the pun.

Graphjam: I can haz BI

Curt Monash — Sun, 16 Nov 2008 21:06:13 +0000

Charts and graphs, from the folks who brought you a whole lot of cute kitten photos:

When people don’t want accurate predictions made about them

Curt Monash — Sun, 16 Nov 2008 17:50:53 +0000

In a recent article on governmental anti-terrorism data mining efforts — and the privacy risks associated with same — The Economist wrote (emphasis mine):

Abdul Bakier, a former official in Jordan’s General Intelligence Department, says that tips to foil data-mining systems are discussed at length on some extremist online forums. Tricks such as calling phone-sex hotlines can help make a profile less suspicious. “The new generation of al-Qaeda is practising all that,” he says.

Well, duh. Terrorists and fraudsters don’t want to be detected. Algorithms that rely on positive evidence of bad intent may work anyway. But if you rely on evidence that shows people are not bad actors, that’s likely to work about as well as Bayesian spam detectors.*

*I.e., pretty much not at all. The idea behind Bayesian spam detectors is that rare words are the best indicator of subject matter. So spammers salt spam with random non-spammy rare words as companions to their spammy ones, and Bayesian filters wave it through. That’s been going on since shortly after I predicted it in 2003 or 2004.

Now let’s take that idea a little further. A lot of data mining and predictive analytics is devoted to figuring out which customers and prospects should get the most attractive offers, or other preferential treatment. The biggest example of this may be telecom companies who invest in reducing churn rates, but other examples abound. For example, gaming companies send pit bosses out to give comp tickets to gamblers who may have reached their pain level on losses. Personalized websites might offer individualized deals as well.

As public awareness of these techniques grows, there’s an obvious risk — consumers could try to game the system so as to get special treatment. It’s not as if this is unknown behavior even without data mining; people complain loudly all the time in hope that they’ll get some sort of mollifying payoff. And so we’ll have just one more reason why data mining models need to be constantly moving targets.

And we two of the many reasons why data mining is an ethical minefield:

It’s tempting for companies to take advantage of their most docile, agreeable, least demanding customers.
It’s tempting for consumers to pretend to hold attitudes different from how they really feel.

Related links

High-performance analytics

Curt Monash — Sat, 15 Nov 2008 19:39:05 +0000

For the past few months, I’ve collected a lot of data points to the effect that high-performance analytics – i.e., beyond straightforward query — is becoming increasingly important. And I’ve written about some of them at length. For example:

MapReduce – controversial or in some cases even disappointing though it may be – has a lot of use cases.
It’s early days, but Netezza and Teradata (and others) are beefing up their geospatial analytic capabilities.
Memory-centric analytics is in the spotlight.

Ack. I can’t decide whether “analytics” should be a singular or plural noun. Thoughts?

Another area that’s come up which I haven‘t blogged about so much is data mining in the database. Data mining accounts for a large part of data warehouse use. The traditional way to do data mining is to extract data from the database and dump it into SAS. But there are problems with this scenario, including:

There’s a lot of data to move.
Therefore it’s tempting to only sample the database rather than analyze the whole thing, which could have at least a slight negative effect on model accuracy.
The result of the process is often some kind of scoring algorithm, and you may want to execute that real-time rather than in batch mode.

Various interesting fixes have been tried.

SAS and Teradata are partnering quite closely to run SAS on Teradata boxes.
Database management system vendors are building at least the data scoring part right into the DBMS. SAS rival SPSS – which relies more on just-in-time SQL and less on batch extracts anyway – reports that hooking into Oracle’s native scoring produces massive performance gains. (To put that another way – I finally got independent confirmation of what Oracle’s Charlie Berger has been telling me for years.)
Data preparation can be handled by the general ELT/ETLT (Extract/(Transform)/Load/Transform – i.e., in-database data transformation) strategies of the data warehouse DBMS vendors.
Oracle (more than most competitors, although SAS/Teradata are headed that way too) actually does all stages of data mining right in the database.

Vendors who are putting considerable marketing emphasis on parallel analytics include:

Greenplum and Aster Data (especially MapReduce)
Oracle (the data mining story and more)
Teradata (the SAS deal, the geospatial effort, and more)
Netezza (especially in connection with the Netezza Developer Network)

I’m sure others would say they belong on the list as well. It’s an important area of competitive differentiation.

Beyond query

Curt Monash — Sat, 15 Nov 2008 17:57:50 +0000

I sometimes describe database management systems as “big SQL interpreters,” because that’s the core of what they do. But it’s not all they do, which is why I describe them as “electronic file clerks” too. File clerks don’t just store and fetch data; they also put a lot of work into neatening, culling, and generally managing the health of their information hoards.

Already 15 years ago, online backup was as big a competitive differentiator in the database wars as any particular SQL execution feature. Security became important in some market segments. Reliability and availability have been important from the getgo. And manageability has been crucial ever since Microsoft lapped Oracle in that regard, back when SQL Server had little else to recommend it except price.*

*Before Oracle10g, the SQL Server vs. Oracle manageability gap was big.

Now data warehousing is demanding the same kinds of infrastructure richness.* When you’re loading data nightly or weekly, and using it to run canned reports, the system can burp for a few hours without anybody getting sacrificially fired. But we’re entering an era of “operational BI,” in which analytics gets integrated tightly into (for example) customer-facing systems, websites and call centers alike. All of a sudden, data warehouses need OLTP-like reliability. Surely not coincidentally, I’ve recently founding myself noting new data warehouse backup strategies from Oracle and Aster Data alike.

*Obviously, there have a few cases where this was needed all along. But my sense is that the numbers are now growing a lot.

In no way do I want to deny that data warehouse performance is crucial. I’m just saying that the other stuff now matters a lot too.

OLTP-like robustness isn’t the only way in which data warehousing issues go “beyond query”; another important subject is high-performance analytics.

The query from hell, and other stories

Curt Monash — Sat, 15 Nov 2008 11:30:58 +0000

I write about a lot of products whose core job boils down to Make queries run fast. Without exception, their vendors tout stories of remarkable performance gains over conventional/incumbent DBMS (reported improvement is usually at least 50-fold, and commonly 100-500+). They further claim at least 2-3X better performance than their close competitors. In making these claims, vendors usually stress that their results come from live customer benchmarks. In few if any of cases, I judge, are they lying outright. So what’s going on?

Multiple things, I think.

Existing data warehouses are often badly optimized. The same technology, configured differently, often would do a much better job.
General-purpose DBMS often require much more tuning for decent complex-query performance than specialized products do. It might have been possible to get that query-from-hell to run fast on the old system, but it wasn’t easy.
Besides, often nobody tried very hard. The value of the query didn’t seem to justify the tuning effort.
Specialized products often really are better for the workloads they’re specialized for.
Different specialized products are best suited for different kinds of analytic workloads.
Different companies send different qualities of benchmark experts to different sales cycles at different times. Smaller vendors with few active sales cycles sometimes actually send their CTOs.
And by the way, vendors do their best to “cook the books.” If one query runs 600X better faster than on the competition, and 19 queries run 2-5X faster, the claim of total speedup is apt to be in the 100X+ range, if that’s defensible by at least one definition of “average.”

Related link

Three bold assertions by Mike Stonebraker

MySQL is being used in an IBM Lotus appliance

Curt Monash — Wed, 12 Nov 2008 06:01:20 +0000

Apparently, IBM is rolling out an appliance for small businesses. MySQL is under the covers. The appliance won’t have a keyboard or monitor, so there won’t be a lot of database administration going on.

Before Solid and solidDB were acquired by IBM, one of the things Solid was proudest of was some embedded apps in which solidDB ran for years in boxes without keyboards or monitors.

I still think it’s a pity that IBM isn’t using solidDB as broadly as the technology deserves. Even so, this is a nice endorsement of MySQL for reliable zero-DBA mid-range use.

Big scientific databases need to be stored somehow

Curt Monash — Fri, 07 Nov 2008 18:36:21 +0000

A year ago, Mike Stonebraker observed that conventional DBMS don’t necessarily do a great job on scientific data, and further pointed out that different kinds of science might call for different data access methods. Even so, some of the largest databases around are scientific ones, and they have to be managed somehow. For example:

Microsoft just put out an overwrought press release. The substance seems to be that Pan-STARRS — a Jim Gray legacy also discussed in an August, 2008 Computerworld article — is adding 1.4 terabytes of image data per night, and one not so new database adds 15 terabytes per year of some kind of computer simulation output used to analyze protein folding. Both run on SQL Server, of course.
Kognitio has an astronomical database too, at Cambridge University, adding 1/2 a terabyte of data per night.
Oracle is used for a McGill University proteonomics database called CellMapBase. A figure of 50 terabytes of “mass storage” is included, which doesn’t include tape backup and so on.
The Large Hadron Collider, once it actually starts functioning, is projected to generate 15 petabytes of data annually, which will be initially stored on tape and then distributed to various computing centers around the world.
Netezza is proud of its ability to serve images and the like quickly, although off the top of my head I’m not thinking of a major customer it has in that area. (But then, if you just sell software, your academic discount can approach 100%; but if like Netezza you have an actual cost of goods sold, that’s not as appealing an option.)

Long-term, I imagine that the most suitable DBMS for these purposes will be MPP systems with strong datatype extensibility — e.g., DB2, PostgreSQL-based Greenplum, PostgreSQL-based Aster nCluster, or maybe Oracle.