Comments on: Sybase IQ technical highlights

By: Mike Stonebraker on “real column stores” | DBMS 2 : DataBase Management System Services

Wed, 12 Jan 2011 13:46:45 +0000

[...] A lesser oopsie is Mike’s criterion “IO-1″, which is written so confusingly that it technically seems not to be met by any of the vendors cited — including Vertica, which introduced Vertica FlexStore in mid-2009. And while I’m at it — Aster Data nCluster definitely meets criterion IO-3; I confirmed that by asking Tasso Agyros. Mike’s “No” for Sybase IQ on his criterion CPU-5 is also pretty questionable, given that Sybase IQ operates on compressed data until “the last possible moment.” [...]

By: Enterprise headlines and summaries, 2009-08-25 « Next Gen Enterprise

Enterprise headlines and summaries, 2009-08-25 « Next Gen Enterprise — Fri, 28 Aug 2009 05:44:43 +0000

[...] Sybase IQ technical highlights General highlights of the Sybase IQ technical story include: * Sybase IQ is an analytic DBMS with a columnar/column-store architecture * Unlike most analytic DBMS, Sybase IQ has a shared-disk architecture. * The Sybase IQ indexing story is a bit complicated, with a bunch of different index kinds. Most are focused on columns with low cardinality, and it least in some cases are a lot like bitmaps. (Sybase IQ when first introduced was a pure bitmap index product, with a single index type “Fast Project”.) But one index kind, “High Group” — designed for columns with high cardinality – is an exception to most generalities about other Sybase IQ index kinds, and instead is more akin to a b-tree. [...]

By: Justin Swanhart

Justin Swanhart — Thu, 27 Aug 2009 21:42:52 +0000

Fastbit doesn’t use dictionaries. It uses Word-Aligned-Hybrid compression (for which they have established an “open” patent) and is available LGPL.

I used it briefly as part of a POC at AdBrite and was VERY impressed. I’ve though about integrating it as a storage engine for MySQL, but MySQL is row engine oriented which sucks for a column store like FastBit.

Eigenbase is pretty cool, and would make a nice frontend to add support for joins and other such SQL constructs which Fastbit currently doesn’t support.

By: Curt Monash

Curt Monash — Thu, 27 Aug 2009 17:06:36 +0000

I haven’t drilled down to the level of detail I’d like on the various Sybase index kinds, so I’ll duck questions like Mark’s.

CAM

By: Julian Hyde

Julian Hyde — Thu, 27 Aug 2009 16:58:16 +0000

Jon, Justin,

Those capabilities are quite straightforward if the database uses dictionary compression and compresses to a fixed number of bits per value. LucidDB does the same in its compressed column stores.

http://www.luciddb.org

By: Justin Swanhart

Justin Swanhart — Wed, 26 Aug 2009 16:40:37 +0000

Jon,

That is possible. An example of such a bitmap index is implemented as Fastbit, an LGPL project.

It implements a type of bitmap compression which supports logical operations between bitmaps w/out decompressing.

You can find it here:
https://sdm.lbl.gov/fastbit/

By: Jon Smirl

Jon Smirl — Wed, 26 Aug 2009 03:25:43 +0000

A couple of years ago there was a TechWave conference where they talked about their ability to do comparison operations on compressed data without decompressing it. I thought that was a cool idea.

By: Justin Swanhart

Justin Swanhart — Tue, 25 Aug 2009 21:59:38 +0000

I’m under the impression that Sybase IQ supports compressed ‘binned’ bitmap indexes. These would require a lot of work to update, though I don’t think they would have to recalculate the entire bitmap.

By: Mark Callaghan

Mark Callaghan — Tue, 25 Aug 2009 16:49:18 +0000

Bitmap indexes in Oracle support high-cardinality columns and their effectiveness in that case is determined by the number of rows per distinct index value, not by the index cardinality. The cost to update them is not as high as you mention for ‘High Group’ indexes in Sybase IQ. Is Sybase IQ storing one bitmap per index value and replacing the entire bitmap on update?

By: Daniel Lemire

Daniel Lemire — Tue, 25 Aug 2009 12:39:35 +0000

Thoughts: I’m guessing that by “projection” they mean bit packing. In any case, it would be really useful if people in this industry agreed to some common definitions, or, minimally, to share their definitions. Reinventing the whole database language for each and every company is painful.