1. I believe so.
2. Seconds.

CAM

Are the numbers with the heading “Old running time” the numbers from the current production environment? I’m wondering if this is another case of comparing new product on new hardware with new vendor supervision/assistance to current product on old hardware w/o vendor assistance.

Are the units minutes or seconds?

I’ll update the post and spreadsheet w/ a geometric option as soon as I can.

Best,

CAM

We’ll make an exception to this if a customer uses another metric (like ‘average query time’)–if that’s what mattered to the customer/evaluator, then who are we to override them.

IT’S ALL ABOUT SLAs…

Another metric we often see, and I think this is what REALLY matters to customers, is related to the service level agreement (SLA) the database application must meet.

It’s the % of queries that run under ‘n’ seconds (or some other unit of time, usually).

In a recent POC, Vertica outperformed a competitive database by 19x (gmqt)–that’s solid, but not very flashy from a marketing perspective.

But the gmqt comparison didn’t matter at all to the prospect. What did matter was that 100% of the queries were answered in under 10 seconds (their performance SLA) vs. 0% for the competitor.

We also see compression (Raw data volume : DB size) measured and compared quite often in POCs. It directly relates to cost of ownership over time.

To sum up, the value of the DBMS to the customer and the motivation to buy it is often based on SLAs (as it should be). Factoring in SLAs (e.g., % of queries that meet SLA) puts the POC results into a context business people will understand (and fund, hopefully).

my 2 cents…

————————–
/
M = n / f_1 * f_2 * … * f_n
\ /
+

This still suffers from the typical problems of a “mean,” but at least it’s suited to the thing being measured.

The need for artificial weighting goes away; this provides a relatively unbiased measure of the “midpoint” of the multipliers.

Of course the *real* improvement is to have a valuation metric tied to what you’re actually using the system for.

Thus, if you’ve got 15 tests, 14 of which found immaterial effects on their runtimes from the tests, and the 15th of which is Truly Essential to run in under 24 hours, where it doesn’t, now, then *really*, the evaluation of goodness properly ought to be almost totally based on that 15th report’s runtime.

Now, that’s TOTALLY context sensitive. If the DW tool spectacularly improves performance on that one report of yours, there is no reason to expect it to have any particular relevant effect on *my* workload.

What we’d like is some way to be able to generalize from performance on your workload to assert something about expected performance on my workload.

Unfortunately, I’m not sure there’s anything other than a POC that can really get at the low level factors that contribute to the actual behaviour. A geometric mean may be better than an arithmetic one by some small margin, but it seems to me you need *way* more parameters than one number/factor in order to do any generalization.