Comments on: Will database compression change the hardware game? http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/ Choices in data management and analysis Fri, 25 Jul 2008 01:36:11 +0000 http://wordpress.org/?v=2.5.1 By: Zizaco http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-50823 Zizaco Wed, 29 Aug 2007 13:23:40 +0000 http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-50823 That increase will be very good for games... Maybe late but it will That increase will be very good for games…
Maybe late but it will

]]> By: On Seaside and State - Cataga http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23176 On Seaside and State - Cataga Tue, 27 Mar 2007 02:25:21 +0000 http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23176 [...] While I love this work I always seem to come away with a couple of nagging issues. One being the amount of memory required to store state on the server for many requests, which over time will become less of an issue. [...] [...] While I love this work I always seem to come away with a couple of nagging issues. One being the amount of memory required to store state on the server for many requests, which over time will become less of an issue. [...]

]]> By: Jay Jakosky http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23095 Jay Jakosky Mon, 26 Mar 2007 08:05:19 +0000 http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23095 I have used QlikView for almost 6 years now and the compression is unbelievable. I routinely get over 20x compression and average 1-2 bytes per field value. I have used QlikView for almost 6 years now and the compression is unbelievable. I routinely get over 20x compression and average 1-2 bytes per field value.

]]> By: David Kanter http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23069 David Kanter Sun, 25 Mar 2007 22:58:48 +0000 http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23069 So, I believe that the throughput/bandwidth aspect of solid state memory is here to stay for a while. Just to satisfy our mathematical curiousity, here's are some stats on flash drives: http://www.bitmicro.com/products_storage_devices.php Most of them look like they provide up to 70MB/s - half the bandwidth of a 15K RPM drive, but ridiculous numbers of IOPs. Most disks top out at 150 IOPS, these provide up to 10x that. The other side of the equation is the tolerance for writes. While flash memory will never be up to par for heavy write workloads, it's possible that future types of solid state memory could deal with 10x more writes. That might be sufficient to deal with many transactional workloads. Also, it's probably good to keep in mind that the MTTF numbers that disk vendors provide are likely not very accurate. I think it would be interesting to compare the TTF for a set of solid state and traditional disk drives. DK So, I believe that the throughput/bandwidth aspect of solid state memory is here to stay for a while. Just to satisfy our mathematical curiousity, here’s are some stats on flash drives:
http://www.bitmicro.com/products_storage_devices.php

Most of them look like they provide up to 70MB/s - half the bandwidth of a 15K RPM drive, but ridiculous numbers of IOPs. Most disks top out at 150 IOPS, these provide up to 10x that.

The other side of the equation is the tolerance for writes. While flash memory will never be up to par for heavy write workloads, it’s possible that future types of solid state memory could deal with 10x more writes. That might be sufficient to deal with many transactional workloads.

Also, it’s probably good to keep in mind that the MTTF numbers that disk vendors provide are likely not very accurate. I think it would be interesting to compare the TTF for a set of solid state and traditional disk drives.

DK

]]> By: Curt Monash http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23007 Curt Monash Sun, 25 Mar 2007 14:03:09 +0000 http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-23007 Thanks, David! But is that a locked-in aspect of the technology, or can we expect it to improve over time? Thanks, David!

But is that a locked-in aspect of the technology, or can we expect it to improve over time?

]]> By: David Kanter http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-22929 David Kanter Sun, 25 Mar 2007 00:38:25 +0000 http://www.dbms2.com/2007/03/24/will-data-compression-change-the-hardware-game/#comment-22929 Hi Curt, Just a quick comment regarding flash. Flash doesn't actually support the bandwidth required for data-warehousing. While flash can support orders of magnitude more IOPS than disk, the bandwidth of each IOP is fairly small. The bandwidth provided is basically (size of read)*(number of IOPS). In comparison, disks provide few IOPS, but under the right circumstances, the read bandwidth becomes very very large. Most flash devices are not going to get any where close to the 70-100MB/s that a disk can achieve (reading data that has high spatial locality). This means flash is largely unsuitable for the underlying data in a warehouse. Flash makes sense for OLTP (actually Texas Memory Systems has devices targeted specifically at OLTP). The only issue there is that many frequent reads will eventually wear down the flash...if you're sustaining thousands of writes/sec, it may prove to be an unwise choice. DK Hi Curt,

Just a quick comment regarding flash.

Flash doesn’t actually support the bandwidth required for data-warehousing. While flash can support orders of magnitude more IOPS than disk, the bandwidth of each IOP is fairly small. The bandwidth provided is basically (size of read)*(number of IOPS). In comparison, disks provide few IOPS, but under the right circumstances, the read bandwidth becomes very very large.
Most flash devices are not going to get any where close to the 70-100MB/s that a disk can achieve (reading data that has high spatial locality). This means flash is largely unsuitable for the underlying data in a warehouse.

Flash makes sense for OLTP (actually Texas Memory Systems has devices targeted specifically at OLTP). The only issue there is that many frequent reads will eventually wear down the flash…if you’re sustaining thousands of writes/sec, it may prove to be an unwise choice.

DK

]]>