I agree with all of that, including points A and B above. In fact, I wrote a white paper about it last year. I do believe, however, that some vendors are working, and have been working, very hard to come up with more durable optimizers to deal with mixed workloads. In the meantime, they must, as you say, make limiting assumptions. I’m not sure there is an alternative, really, it’s just the degree of limiting involved.

Sanjay’s point is well-taken. M-P’ing SQL Server isn’t going to address this problem. I suspect there is more to this than we’ve heard so far. When IBM brought out the SP2, we were pretty hopeful since DB2 had a pretty good optimizer (compared to its competition) and so did Informix, but we never got the performance we hoped for. Far from it.

Any fool can build a 100Tb data warehouse, and many have. It isn’t hard to throw a hundred processors at it and get “linear” scale up, so long as you don’t run more than one query at a time. But getting something like that to perform in a real operation, with bursty, bunched up and mixed query types is an NP hard problem.

I agree with much more of that than I disagree with. Most of the Very Large warehouses tend to actually be data marts, simpler than some of the smaller ones. So being able to perform decently at a low price on a typical 100 TB+ “data warehouse” is no proof of ability to perform well on all 2-5 TB warehouses.

I don’t know of any data warehouse product that seems to truly be optimized for general, complex queries. Instead, they use one or both of two main techniques:

A. Making limiting assumptions about the nature of the queries, and exploiting those for great performance.

B. Striving to be superfast on the more elementary queries complex ones ultimately decompose to, and hoping this ripples through to decent performance on everything the user needs.

First you say that nobody’s debating SMP vs. MPP any more, even though that’s a major topic of this blog, which sort of equates to you calling me a nobody. Then you say you’re not really talking about the technology after all, but rather “practice”, whatever that is that’s separate from BI (and from things like data governance, which if I recall correctly you were also objecting to).

If you took the trouble to be remotely clear, I might or might not agree with your opinion. As matters stand, you’re right — I have no argument with what you’re saying. In fact, I’ve wasted altogether too much time trying to puzzle it out.

CAM

So I’ll interpret that as indicating you have no argument. Data warehousing is NOT becoming commoditized…because you said so.

I’ll look elsewhere for enlightenment.

-NR

Considering that your comments seem to be highly critical of, and also to mischaracterize, blogs like this one, I’ll live at peace if you take some offense at my reply.

CAM

I don’t know how to answer that question “in any form.” Perhaps there are peripheral forms I’m not aware of. But I think I’ve made myself abundantly clear. The practice of data warehousing is largely formulaic. I tried to draw the distinction between the industry and the practice in response to Phil’s comments. Unfortunately, blogs that discuss the industry, not the practice, get undue attention. But then, so do reality shows.

BI is its own big mess, I shouldn’t have commingled the two.

Also, I guess I take some offense at your use of the term “complaining.”

-NR

На мой взгляд пока как то незаметно в российском инете прошло достаточно важно событие – Microsoft объявил…

Is it really data warehousing in any form that you’re complaining about, or is it BI?

CAM