Comments on: Yahoo wants to do decapetabyte-scale data warehousing in Hadoop

By: Bioinformatics and mythology. You still need to manage the data

Bioinformatics and mythology. You still need to manage the data — Thu, 10 Dec 2009 03:43:30 +0000

[...] Yahoo’s wants to do decapetabyte-scale data warehousing in Hadoop (dbms2.com) [...]

By: Mark Tsimelzon

Mark Tsimelzon — Thu, 01 Oct 2009 23:19:51 +0000

Curt, you may want to add that Yahoo’s Hadoop team is growing fast! If anybody wants to join us, we are looking for developers, architects, testers, and managers: http://developer.yahoo.net/blogs/hadoop/2009/10/do_you_have_what_it_takes_to_j.html

By: Curt Monash

Curt Monash — Thu, 01 Oct 2009 22:37:29 +0000

Everest = Yahoo’s proprietary Postgres-based column store that is managing petabytes of data.

By: Jerome Pineau

Jerome Pineau — Thu, 01 Oct 2009 20:23:12 +0000

Besides Oracle and their internal (Everest) engine you mentioned I mean (am assuming it’s not all ORCL is it?)
The video is really full of interesting stuff. I wonder how many “nodes” they work with and what kind of fabric is used and how these things are clustered . Isnt Xen the same VM EC2 is using?

Fascinating stuff.

By: Curt Monash

Curt Monash — Thu, 01 Oct 2009 20:17:47 +0000

Jerome,

As per other threads, it’s clear Yahoo is using quite a bit of Oracle.

Otherwise, I couldn’t say.

By: Jerome Pineau

Jerome Pineau — Thu, 01 Oct 2009 20:03:23 +0000

Curt, I’m curious: how much data is Yahoo currently managing total and do they use commercial RDBMSs at all and if so which ones?
Thanks.