In short, most joins involve sorts and merges, which are expensive and can get very expensive when large data sets are involved. There are only two join plans that provide linear scalability, the hash join and the hash merge join. The hash merge join requires a hash based file sytem (different from hash distribution). The hash join employs a similiar technique but in memory. The problem is that memory runs out quickly and is often used for other operations like buffering. Teradata is the only RDBMS that provides the hash merge join.

So if I have to sort and merge large data sets, I really want the hash merge join available to the optimzer.

I have a design that will ensure SUB-linear scalability, up to over a petabyte. On one terabyte of data, I’ll throttle performance by a factor of 10. On four terabytes, I’ll throttle it only by a factor of 8 … OK, I’m kidding. But to compare constant_1 times n vs. constant_2 times nlogn, it’s interesting to know what constant_1 and constant_2 are.

More generally, I’m confused by what you’re saying. You seem to be assigning a single scalability function to all join plans on a particular product, no matter what strategy the particular query’s execution plan uses. Taken literally, that’s totally absurd, and I’m not guessing successfully at your actual and surely more sensible meaning.

Linear scalability is very rare in computing and I’d be interested in knowing if anyone besides Teradata provides it in their RDBMS.

Most of the row-based competitors can, as one implementation option, do a hash partition, forgo indexes, and expect the queries to be satisfied by table scans.

So I’m not clear as to exactly what architectural point you are making that puts Teradata ahead of the newer guys, or for that matter that makes it impossible to use Oracle in the way that you described.

If all you’re saying is that b-trees aren’t the way to do decision support, and that the architectures of specialty products reflects this fact better than Oracle’s does, I agree completely. But it looked as if you were going to an extreme that I don’t see the foundation for.

CAM

Erin McCabe recently joined HP’s BI unit. Expect better PR from them in the future!

Best,

CAM

CAM

“Expressions – I assume this means projects and selects – are done via a kind of byte code, on the CPU. Greg suggested Teradata uses a similar approach.” Actually this pertains to the computation of any expression value in the engine, including aggregate functions, arithmetic functions, string functions, and so on. The idea behind using a byte-code machine is that the machine can, in principle, be “compiled” (optimized) at query build time to eliminate code that is unnecessary for this computation in this particular context. Other systems, including Sybase SQL Anywhere, use this approach.

“Neoview’s Cascades-based optimizer seems to be smart enough to, for example, do aggregations before joins when it makes sense. (Much the same is true of Aster Data’s optimizer.)” – Pushing/pulling aggregation above/below a join was studied by a fellow graduate student, Paul Yan, at the University of Waterloo in the mid-1990s as part of his PhD thesis (under the direction of Paul Larson, now at Microsoft Research). As far as I know DB2 was the first product to incorporate these optimizations.

“By the way, the basic idea behind Cascades – or at least Neoview’s version of it — is that it uses more heuristics than conventional cost-based optimizers do. That it, Neoview starts out with a candidate plan – perhaps derived in the usual way – and then considers variants on it.” – It is difficult to know how much Neoview’s implementation differs from other transformation-based optimizer implementations (such as Microsoft SQL Server) based on the Cascades framework (originally developed by Goetz Graefe, now at HP Labs). Every optimizer uses heuristics to reduce the size of the search space; whether or not one uses “more” heuristics over another is difficult to assess, because those assumptions are rarely documented, if even made public.