People who think current “relational” DBMS aren’t relational enough often suggest that’s the kind of foundation DBMS should have. And unlike Required Technologies’ TransRelational (TM) shtick, ScaleDB’s really is an OLTP-oriented approach.

1. Different DBMS should be used for different purposes. I am in violent agreement with that point, which is indeed a major theme of this blog.

2. Vertica’s software is 50X faster than anything non-columnar and 10X faster than anything columnar. Now, some of these stats surely come from the syndrome of comparing the future release of your product, as tuned by world’s greatest experts on it who also hope to get rich on their stock options in your company, vs. some well-established production release of your competitors’ products, tuned to an unknown level of excellence,* with the whole thing running test queries that you, in your impartial wisdom, deem representative of user needs. Or something like that …

*For example, were Vertica’s competitors set up with vertical partitioning?

That said, is the implicit claim that columns are 5X faster than rows for data warehouse queries fundamentally ridiculous? For data retrieval, probably not. The improvement obviously varies hugely depending on how wide a table is and what fraction of the columns are needed in a particular query, but 5X doesn’t sound hopelessly out of whack. But when we focus on the joins themselves — well, I’ll confess to not knowing quite enough about current DBMS designs to judge fully, but it sounds a bit fishy to me. I think row-based DBMS are pretty stupid about, during joins, carting around all the columns that aren’t involved in the join criteria. But the extent to which that extra baggage actually gets in the way of efficient processing doesn’t seem as clear.

To look at it another way: If columns were really all THAT great, mightn’t existing bitmap capabilities — including those of Oracle et al. — be more widely used?

3. (Different) columnar systems can and should be developed that conquer the world and take over most or all data management niches. At least, that’s what he seemed to be suggesting. It’s not a totally crazy idea. After all, text indexes are essentially columnar systems, as were the stellar pre-relational “inverted list” DBMS Adabas, Datacom/DB, and Model 204. And what is abstract datatype support other than specialized handling on a column-by-column basis?

Note: The last time I suggested to Mike the idea that what the world really needed was an extensible columnar system that used different data access methods for different datatypes in different columns, he didn’t seem to have thought about the subject a whole heck of a lot. But I continue to think that’s a very interesting direction for further work.

But can the claim be stretched so far as to suggest that columnar systems are the best way to go about OLTP? That WAS exactly what was claimed for Required Technologies, Inc.* and the “TransRelational” model. But I spent a whole lot of cycles on, as it were, debunking that tripe. More generally, I think the most natural OLTP architectures are those that keep things together that get updated and retrieved together, whether these are relational rows, “objects” in the strict OO sense of the term, or something else. So at least in its strongest form, I think this third claim reflects a bit of, er, poetic license on Mike’s part.

*Mike actually was involved with RTI. However, he does not seem to be responsible for any of the mishegas that surrounded it. Indeed, he seems to have given them some pretty sound operational advice back when they were still a viable entity.

As a plausibility argument for the benefits of columnar systems, Mike likes to trot out three “tick processing” software offerings (that’s in the stock price sense of “tick”) – Vhayu’s Velocity, Sungard’s FAME, and Kx’s kdb. I know even less about these than I do about SAND, but I’m conjecturing they’re time series data stores rather than full RDBMS. Mike is less eager to talk about Required Technologies, a failed columnar RDBMS startup that he was involved in, and which is the pretext for (through no fault of his) TransRelational hype.

If columnar systems take off, be prepared for a lot of “We do that too” claims from mainstream DBMS vendors. Oracle and other conventional relational DBMS do feature bit-mapped indices. But while they’re more versatile than pure columnar systems, they won’t do columnar things as efficiently as the purists (or near-purists) do. Full-text indices are closely akin to bit-maps– e.g., SAP’s BI Accelerator grew out of TREX – but obviously it would not be accurate to call them “columnar” (or, for that matter, “relational”).

Old way: Assume I/O is random; lower total execution time by improving selectivity and thus lowering the amount of I/O.

New way: Drive the amount of random I/O to near zero, and do as much sequential I/O as necessary to achieve this goal.

Examples include:

• Data warehouse appliances (see especially this discussion of DATallegro’s architecture)
• Columnar systems (see Nathan Myer’s first comment in this discussion of the much-hyped Required Technologies prototype)
• Memory-centric systems, notably SAP’s BI Accelerator

The hardware logic is compelling, as long as we rely on hard disks rather than, say, flash memory. Rotation speed has only gone up 12.5-fold in the entire 50-year history of the hard drive, and currently maxes out at 15,000 RPM, which puts a floor of 2 ms on average random access time. But streaming data on and off disk gets exponentially faster, in line with increases in disk density and semiconductor performance. Hence sequential data access gets ever faster, while random access does not.

What I don’t 100% understand yet, however, is the full array of techniques used by the traditional leaders to co-opt or combat this trend. I’m looking into that; in particular, I have a call scheduled with Oracle.

I hope to write about this issue in my October Computerworld column. (My columns are typically submitted on the first Monday or Tuesday morning of the month, to appear in the following week’s edition.) Or if it slips from October, then soon thereafter. Any thoughts in the interim would be most welcome.

That said, if you aren’t amused by flame wars, it’s probably not worth the trouble to read.

Alf did poke through the dreck, however, to make a reasonable challenge, which can be paraphrased as:

OK.  Suppose you’re right that no implementation has ever provided evidence of TransRelational’s usefulness for building a True Relational DBMS. It’s still theoretically fascinating.

My response was as follows:

Here are two big problems with TransRelational that are perfectly theoretical.
First, it assumes that values can be concisely stated, presumably as numbers or character strings. That isn’t a good match to complex datatypes such as, say, documents that should be full-text indexed.

Second, it assumes that there’s a natural sort order. That could be a bit of a problem even for, say, geospatial. One would think there’s a workaround in the geospatial case, e.g. like Oracle’s old hhencode. But hhencode was a fiasco, I think because it didn’t actually measure proximity very effectively.

Admittedly, both of my objections also apply to good old b-trees. Still, they speak against the potential of a TransRelational implementation to achieve the kind of generality I think modern applications do and will increasingly demand.

Basically, I think a “True Relational” DBMS that was only useful for columns with natural sort orders wouldn’t be particularly interesting. And “The Third Manifesto” notwithstanding, that’s the only kind anybody seems to have even hinted at trying to bring to market.

1. While the company indeed started in New York, before it shut down in November/December 2002 it had moved to California. The reason was the availability of DBMS engineers. That makes sense.

2. The company had venture capital lined up to keep it going, but the deal fell apart due to “governance” issues. I.e., the investors wanted to bring in a new CEO and, as they always do, control the board. Rather than acceding to this common if annoying demand, the CEO shut the company down.

Incidentally, my contact says most of the developers went to Google, and most of the rest went to Yahoo.

3. The product never had a DML or DDL other than SQL. Any claims that it could implement a “true relational” language are speculative. Any claims that it already did implement such a language are utter falsehoods, unless a whole lot of secret development has been done (for which there was no money) without the shareholders being notified.

4. My contact insists that, for what it was, the product was ready for beta release when the company shut down. It was an analytics-oriented DBMS, definitely not designed for update-heavy applications. Indeed, he explicitly said that it was for “decision support, with batch loads into the database.”

5. In their tests, it outperformed Oracle on most queries. These tests seem to have been mainly performed on a database in the 100 gigabyte size range. It had better than a 10X advantage in storage space vs. a highly denormalized star-schema approach. However, claims of “orders of magnitude” performance improvements are utterly unfounded.

6. My contact viewed the TransRelational architecture as being memory-centric, but in the actual product they tried to marry concepts of TransRelational with efficient storage on disk. He gave two examples of the kinds of features they added. One is the ability to sort the data for efficient I/O. Another is to in certain cases reaggregrate the data. I don’t have further detail than that.

7. The core idea of how the product worked is that it stored, for each column, a unique list of the values, and also the frequency of their appearance. (Called “standard distribution values”). Fabian Pascal’s denial that this was a columnar data store is bullshit. It also stored lots of indices to reconstruct every possible record. Claims that there are no indices are bullshit. The overhead of reconstructing records in memory was negligible.

Bottom line: This was a very cleverly designed product, that might have made an impact in the market for data warehousing DBMS. But it is very far from what it is being promoted as being.

I’m sorry for the nasty tone of my posts on this subject, but this product has been consistently misrepresented. Those misrepresentations have been used by unpleasant people as hammers in flame wars, and they also seem to be the basis for separating unsuspecting seminar attendees from considerable amounts of money. A lot of time and effort has been spent debunking this BS, including by me. Any honest, knowledgeable person should have stopped promoting it years ago.

Even odder is Date’s multi-paragraph diatribe about a specific two-word phrase I supposedly used. Maybe if he’d looked and read what I’d really said, it would have made more sense to him. Or maybe not. He also bragged about not knowing who I am, thus revealing a couple of things. First, he can’t be bothered to use Google any more than he can be to actually read a blog post he’s criticizing. Second, he’s pretty out of touch with the actual DBMS industry, something which can be independently inferred from his frequently bizarre statements about DBMS vendors and their technology. (Chris Date may still be one of the world’s great experts on the use of DBMS, but when it comes to actually building them, he seems to be pathetically ignorant.)

Other things he says in that piece, however, are sufficiently coherent to warrant an attempt at response. So here goes.

He starts by asking, in effect, “Why are you picking on me?” Well, I didn’t start out by intending to pick on Chris Date. But Fabian Pascal engaged me in flaming discussion, and when he found himself out of his depth, quickly resorted to what he seemingly thought was a discussion-ender, namely dropping the sanctified name of “Chris Date.” So I did a little poking around, and discovered that Date indeed has explicitly endorsed Pascal and Pascal’s website, posts on Pascal’s website “on a fairly regular basis,” lets Pascal speak for him, and generally seems in my opinion to be responsible for and indeed to apparently agree with the views Pascal ascribes to him.

Beyond that, I’m picking on Date because he is misleading his paying audience(s) on the subject of TransRelational technology. In that regard he’s acting like so many other DBMS vendor marketing spokespeople, who may not have been precisely lying, in that they probably deluded themselves before peddling nonsense to others. But at least John Cullinane and Dave Peterschmidt were shipping actual, useful products. Nor did they charge for seminar admittances or book sales just so that people could hear their pitches.

Moving on, Date claims that his views are scientific in general nature. However, he ignores overwhelming evidence against them, namely the last 20 years of development and use of relational DBMS, and presents almost no empirical evidence for them. So yes, I stand by my claim that the pure-relational fanatics are “reasoning” in a quasi-religious manner much more than they are reasoning scientifically (in any sense of “science”).

This ties into the claims that to be against relational theory is to be against logic, that relational theory is based on science that was established over 2,000 years ago, and so on. The most outrageous versions I’ve seen of such hokum indeed came from Pascal’s keyboard rather than Date’s. But as noted above, Chris Date supports Fabian Pascal, and anyhow what Date himself says is bad enough.

About the most expansive valid claim that can be made along those lines is “Sound mathematical predictions can be made about the behavior of systems built in conformance to the Relational Model, and the same is not at this time equally true of systems that do not conform to the RM. In particular, it is possible to make mathematically sound assurances about data integrity.” That’s a strong argument for using the RM in certain contexts – quite a few contexts, actually. But it hardly refutes the central claims of the DBMS2 agenda, nor is it a strong argument against the use of today’s SQL-oriented DBMS. And it certainly doesn’t excuse the bitter insults Date and Pascal regularly hurl at both the DBMS user and DBMS developer communities. Nor does it justify their vitriolic attacks on me. Nor does it support many other things that Date and/or Pascal have said.

I think this covers most of what Date said in his screed. Oh, he had a lot more numbered points than that, but they seemed to repeatedly touch on the issues mentioned above. As for the less personal, and much more important, aspects of our disagreement – well, this whole blog repeatedly covers those subjects.

Finally, a bit of self-examination here – is my disagreement with Date really that important, or am I like Captain Queeg with the missing strawberries, trying to reenact the successes of my younger years (e.g., Cullinet and Sybase)? Well, Required Technologies surely is not an important target – but nor have I focused on it. My real target is any claim that a large enterprise should obsessively try to run itself on One Grand Centralized relational database (especially one that actually ships commercially; I’ve been assuming that no enterprise would try to bet the farm on an idealized “true relational” product that lacks the key buying criterion of “it actually exists.”). It just so happens that the shrillest initial opposition to this idea came from Chris Date’s spokesman Fabian Pascal, and it’s Pascal who brought Date into the conversation.

This is regrettable on multiple levels.

1. Required Technologies shut down product development in 2002, after running through $30 million; there’s great acrimony between investors and the CEO; and lawsuits are likely.

2. Required’s product never did most of what Date seems to be claiming it now does. It was a read-oriented columnar data store, much like Sybase IQ or a number of other products from younger companies.

The basic idea behind these products is like that of bitmapped indices, even if the actual implementation doesn’t use literal bitmaps, or indeed indices at all. To be competitive they generally need three things – good compression, decent updating, and a way to handle higher-cardinality columns. (Cases in which cardinality is low enough for true bitmapping to make sense are the exception, not the rule.) These products may also exploit the fact that a bitmapped index recreates all the information in the database, and hence you don’t have to keep two copies; the index effectively BECOMES the database. (Thus, Required’s claim not to have indices at all is one of the few parts of its story that is NOT ridiculous.)

This general product design is not all bad. Yes, it destroys update performance, but there are potential techniques to keep that problem bounded (e.g., with update staging tables that have a traditional row-based architecture). It seems to assume that you primarily want to store datatypes with concise values and a natural sort order (e.g. numbers, character strings), but other architectures share that flaw, and up to a point it could use the same workarounds they do. And it certainly can show much better query performance than a traditional row-oriented database, especially if the traditional database is poorly designed or deliberately unoptimized.

However, there’s no way that an essentially dead company can complete what would be a commercially useful product in this area. Unless a lot changes – a lot more than Date is evidently implying – Required isn’t going to be shipping something that can compete effectively with standard big-name DBMS products.

Date and Pascal should be deeply ashamed of themselves.

Some of the links below have references to “orders of magnitude” performance improvements over conventional systems, perhaps based on benchmarks. Even if that’s true, it proves very little. A benchmark of a prototype system shows almost nothing about performance in real world production situations.

Related links

Edit: A number of these have gone broken since the time of the original post

• Required’s phantom website – no content, except a West Coast address. Even the email contact form is busted.
• Required’s phantom site in 2003, per archive.org – same thing
• The company’s website in 2002 — it was located on the East Coast then
• An article perpetuating the myth – from the notoriously hype-filled dbdebunk.com
• Another article on the same site – this one is written by highly imaginative site owner Fabian Pascal himself
• DMOZ’s links for Required – patents and a detailed resume from the VP of Development
• A job ad from when Required actually was doing substantial development
• Date is apparently pitching this dream next week; presumably some portion of the hefty attendance fee can be ascribed to the day-long “privilege” of hearing this sales pitch
• Date’s financial interest in the company, as per yet another dbdebunk.com article, retracting a prior denial by the unreliable Mr. Pascal. But even the corrected version of the story probably understates the case. It says Date served on Required’s scientific advisory board on a “volunteer” basis. Much more likely is that he got stock or stock options.

Note: I am NOT under the delusion that the company stuck with the in-memory architecture it pitched to its inventors and disclosed in its patent, and which was the basis for an online debate about the disk-worthiness of the architecture within the past year. I’m relying on nonpublic — but thoroughly reliable — sources. When a company lays off that many people, it’s no big trick to, after the fact, figure out what it was doing.