Comments on: Facebook’s experiences with compression http://www.dbms2.com/2009/05/14/facebooks-experiences-with-compression/ Choices in data management and analysis Thu, 09 Feb 2012 16:57:09 +0000 hourly 1 http://wordpress.org/?v=3.0.3 By: Ashish Thusoo http://www.dbms2.com/2009/05/14/facebooks-experiences-with-compression/#comment-121924 Ashish Thusoo Mon, 18 May 2009 04:53:18 +0000 http://www.dbms2.com/?p=783#comment-121924 Ivan, Memory usage is definitely an issue and that typically means that we would not be able to run as many map/reduce slots in the cluster. We are targeting this mostly for archival at this point and there the latency requirements on decompressing or compressing this data is not that high yet. Ivan,

Memory usage is definitely an issue and that typically means that we would not be able to run as many map/reduce slots in the cluster.

We are targeting this mostly for archival at this point and there the latency requirements on decompressing or compressing this data is not that high yet.

]]> By: Ivan Novick http://www.dbms2.com/2009/05/14/facebooks-experiences-with-compression/#comment-121534 Ivan Novick Thu, 14 May 2009 17:12:41 +0000 http://www.dbms2.com/?p=783#comment-121534 Also be careful about the memory usage of bzip2, which may be prohibitively high on large data sets. On gzip there are 9 different compression levels, level 6 seems to give the best balance between cost and data size. 6X seems about right for web log data Also be careful about the memory usage of bzip2, which may be prohibitively high on large data sets.

On gzip there are 9 different compression levels, level 6 seems to give the best balance between cost and data size. 6X seems about right for web log data

]]>