May 24, 2008

DATAllegro on compression

DATAllegro CEO Stuart Frost has been blogging quite a bit recently (and not before time!). A couple of his posts have touched on compression. In one he gave actual numbers for compression, namely:

DATAllegro compresses between 2:1 and 6:1 depending on the content of the rows, whereas column-oriented systems claim 4:1 to 10:1.

In another recent post, Stuart touched on architecture, saying:

Due to the way our compression code works, DATAllegro’s current products are optimized for performance under heavy concurrency. The end result is that we don’t use the full power of the platform when running one query at a time.

Not immediately seeing the connection, I emailed Stuart for clarification. His answer boiled down to:

More precisely, he wrote:

This gets a little complex, but decompression effectively adds latency to I/O. Our platform runs a certain number of threads on each server for a given query (currently six) so that each CPU has a dedicated disk platter. With a single query, the (serial) latency for decomp after the I/O effectively creates some I/O wait time and therefore slows down overall performance. With multiple queries, this I/O wait time is used for other useful work on different queries, so the platform is fully utilized.

Also, we have eight cores, so six threads doesn’t always use all of them for a single query. Again, this goes away when you have multiple queries.

Since most customers have multiple queries running at the same time, we optimized for that workload, rather than trying to run more threads for single queries, because the latter starts to create problems when you have LOTS of things going on.


Leave a Reply

Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:


Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.