http://www.youtube.com/watch?v=yrLd-3lnZ58

Cool stuff,
E.

One thing to note is that the opinion of working on compressed data sets is mostly useful for the major ordering columns only refers to the RLE compression. Like you write, in cases with large domain cardinality RLE won’t do much for non-sorted data.

Still, other forms of compression can be used and data compressed with those can be analyzed without decompressing, see e.g. http://scholar.google.com/scholar?q=%22The+Implementation+and+Performance+of+Compressed+Databases.%22

m.

I edited in two corrections (Ph.D, CPU cycles).

Best,

CAM

Thank you for a nice writeup on VectorWise. While generally correct, here are some clarifications:

- the VectorWise technology belongs fully to our company (no academic institution, including CWI, can control it)

- the MonetDB open-source system originated from the PhD research of Peter Boncz under supervision of Martin Kersten, while the VectorWise database engine is a technology generation later and came out of my own PhD (not MSc) research, supervised in turn by Peter Boncz. Other CWI group members also have significant contributions to both projects.

- we do hope to make VectorWise technology available as early as possible, and 2010 is very possible, but please do not treat it as an official plan

- as for the string compression, we use something called PDICT, which is a new – outlier resistant – form of dictionary encoding.

- like you wrote, the main thing about the compression methods in VectorWise is that they are much faster than existing methods. As for the performance, we take a few “CPU cycles” (not “steps”) for one element. Links to publications with more technical info can be found on: http://www.vectorwise.com/index_js.php?page=company_origins

- the place to visit for more info on the Ingres VectorWise project is http://www.ingres.com/vectorwise

Best regards,
Marcin Zukowski

If your table has few dimensions, this makes no sense. But for high dimensional tables, it rings true. Indeed, columnar compression often comes through run-length encoding (RLE), after sorting (lexicographically). Yet, only the first few columns (in sorting order) will end up compressible by RLE after sorting them.

See for example:

Daniel Lemire, Owen Kaser, Kamel Aouiche, Sorting improves word-aligned bitmap indexes. Data & Knowledge Engineering (to appear).
http://arxiv.org/abs/0901.3751
http://www.slideshare.net/lemire/all-about-bitmap-indexes-and-sorting-them

This suggests that they are not relying much on RLE. It might be that vector processing does not work well in conjunction with RLE?