Comments on: MapReduce for data mining? Maybe for variable-schema analytics. http://www.dbms2.com/2008/01/19/mapreduce-variable-schema-analytics/ Choices in data management and analysis Sat, 31 Jul 2010 00:21:55 -0400 http://wordpress.org/?v=2.8.4 hourly 1 By: More Google reliability woes | DBMS2 -- DataBase Management System Services http://www.dbms2.com/2008/01/19/mapreduce-variable-schema-analytics/#comment-95582 More Google reliability woes | DBMS2 -- DataBase Management System Services Mon, 25 Aug 2008 07:50:44 +0000 http://www.dbms2.com/2008/01/19/mapreduce-variable-schema-analytics/#comment-95582 [...] reliability issues are ever worse. As I previously pointed out, this is evidence against the notion that MapReduce is a replacement for established DBMS. Share: These icons link to [...] [...] reliability issues are ever worse. As I previously pointed out, this is evidence against the notion that MapReduce is a replacement for established DBMS. Share: These icons link to [...]

]]> By: Stuart Frost http://www.dbms2.com/2008/01/19/mapreduce-variable-schema-analytics/#comment-68632 Stuart Frost Tue, 22 Jan 2008 00:29:32 +0000 http://www.dbms2.com/2008/01/19/mapreduce-variable-schema-analytics/#comment-68632 This is all interesting on a number of fronts. First of all, the critique of MapReduce by DeWitt and Stonebraker is breathtakingly arrogant. MapReduce was clearly not designed to solve the same problems as an RDBMS, so it's strange to criticize it for not having the same functionality. As for the comment that MapReduce will be difficult to scale - well, it's hard to argue with 20PB per day! Google's benchmarks are also pretty revealing. Using 1,800 servers to grep through 1TB of data in 2.5 mins is incredibly inefficient. Using user defined functions (UDFs) in one of our appliances, I estimate that we'd get through the same amount of work on less than 16 nodes - maybe as few as eight, given the likelihood of higher than normal compression ratios. Not sure about how fast the sort would run on our appliances, but over 800s to sort 1TB on 1,800 servers also seems very, very slow - as do the I/O rates shown on the charts. Seems like they are just throwing an awful lot of hardware at the problem - don't tell Al Gore! Stuart Frost CEO, DATAllegro This is all interesting on a number of fronts.

First of all, the critique of MapReduce by DeWitt and Stonebraker is breathtakingly arrogant. MapReduce was clearly not designed to solve the same problems as an RDBMS, so it’s strange to criticize it for not having the same functionality. As for the comment that MapReduce will be difficult to scale – well, it’s hard to argue with 20PB per day!

Google’s benchmarks are also pretty revealing. Using 1,800 servers to grep through 1TB of data in 2.5 mins is incredibly inefficient. Using user defined functions (UDFs) in one of our appliances, I estimate that we’d get through the same amount of work on less than 16 nodes – maybe as few as eight, given the likelihood of higher than normal compression ratios. Not sure about how fast the sort would run on our appliances, but over 800s to sort 1TB on 1,800 servers also seems very, very slow – as do the I/O rates shown on the charts.

Seems like they are just throwing an awful lot of hardware at the problem – don’t tell Al Gore!

Stuart Frost
CEO, DATAllegro

]]>