Database compression is heavily affected by the kind of data
I’ve written often of how different kinds or brands of data warehouse DBMS get very different compression figures. But I haven’t focused enough on how much compression figures can vary among different kinds of data. This was really brought home to me when Vertica told me that web analytics/clickstream data can often be compressed 60X in Vertica, while at the other extreme — some kind of floating point data, whose details I forget for now — they could only do 2.5X. Edit: Vertica has now posted much more accurate versions of those numbers. Infobright’s 30X compression reference at TradeDoubler seems to be for a clickstream-type app. Greenplum’s customer getting 7.5X — high for a row-based system — is managing clickstream data and related stuff. Bottom line:
When evaluating compression ratios — especially large ones — it is wise to inquire about the nature of the data.
Comments
4 Responses to “Database compression is heavily affected by the kind of data”
Leave a Reply
Curt,
I’d love to get a better definition of compression. When they say 60X compression, do they really mean 60x of all of the information with no data loss? In reality, weblog and clickstream data is full of junk that doesn’t get integrated. I haven’t done this myself in a few years, but I seem to recall that we stripped out and analyzed only a small faction of each record.
-NR
Neil,
I’m pretty sure it’s lossless. That’s always what’s meant in talking of compression of other kinds of data, and I don’t know why this kind would be an exception.
CAM
[…] It’s clear what Omer means by most of those categories from reading the post, but I’m a little fuzzy on what “Consumer Data” or “Marketing Analytics” comprise in his taxonomy. Anyhow, Omer’s post is a huge improvement over my recent one — based on a conversation with Omer — which featured some far less accurate or complete compression numbers. […]
[…] Сжатие данных. Я много писал о компрессии данных. […]