Comments on: Data warehouse load speeds in the spotlight http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/ Choices in data management and analysis Thu, 09 Feb 2012 16:57:09 +0000 hourly 1 http://wordpress.org/?v=3.0.3 By: While I’m venting about benchmarks | DBMS2 -- DataBase Management System Services http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-129665 While I’m venting about benchmarks | DBMS2 -- DataBase Management System Services Wed, 08 Jul 2009 23:58:50 +0000 http://www.dbms2.com/?p=634#comment-129665 [...] last year, Vertica made hoo-hah about what it called a world-record data warehouse load speed benchmark.  I wrote at the time that this showed Vertica wasn’t painfully slow at loading, always a [...] [...] last year, Vertica made hoo-hah about what it called a world-record data warehouse load speed benchmark.  I wrote at the time that this showed Vertica wasn’t painfully slow at loading, always a [...]

]]> By: Greenplum claims very fast load speeds, and Fox still throws away most of its MySpace data | DBMS2 -- DataBase Management System Services http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-113839 Greenplum claims very fast load speeds, and Fox still throws away most of its MySpace data | DBMS2 -- DataBase Management System Services Fri, 20 Mar 2009 09:10:49 +0000 http://www.dbms2.com/?p=634#comment-113839 [...] warehouse load speeds are a contentious issue.  Vertica contrived a benchmark with a 5 1/2 terabyte/hour load rate.  Oracle has gotten dinged for very low load speeds, which then are hotly debated.  I was told [...] [...] warehouse load speeds are a contentious issue.  Vertica contrived a benchmark with a 5 1/2 terabyte/hour load rate.  Oracle has gotten dinged for very low load speeds, which then are hotly debated.  I was told [...]

]]> By: Expressor pre-announces a data loading benchmark leapfrog | DBMS2 -- DataBase Management System Services http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-106250 Expressor pre-announces a data loading benchmark leapfrog | DBMS2 -- DataBase Management System Services Sun, 04 Jan 2009 18:22:15 +0000 http://www.dbms2.com/?p=634#comment-106250 [...] Software plans to blow the Vertica/Syncsort “benchmark” out of the water, to wit What I know already is that our numbers will between 7 and 8 min to load [...] [...] Software plans to blow the Vertica/Syncsort “benchmark” out of the water, to wit What I know already is that our numbers will between 7 and 8 min to load [...]

]]> By: cooper to logan » Blog Archive » the white zone if for immediate loading only http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-106240 cooper to logan » Blog Archive » the white zone if for immediate loading only Sun, 04 Jan 2009 14:58:16 +0000 http://www.dbms2.com/?p=634#comment-106240 [...] There has been a renewed buzz in the data integration vendor world around the coveted tpc-h benchmark. A discussion about the latest can be found at dbms2. [...] [...] There has been a renewed buzz in the data integration vendor world around the coveted tpc-h benchmark. A discussion about the latest can be found at dbms2. [...]

]]> By: More from Vertica on data warehouse load speeds | DBMS2 -- DataBase Management System Services http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-106166 More from Vertica on data warehouse load speeds | DBMS2 -- DataBase Management System Services Sat, 03 Jan 2009 05:37:39 +0000 http://www.dbms2.com/?p=634#comment-106166 [...] month, when Vertica releases its “benchmark” of data warehouse load speeds, I didn’t realize it had previously released some actual customer-experience load rates as [...] [...] month, when Vertica releases its “benchmark” of data warehouse load speeds, I didn’t realize it had previously released some actual customer-experience load rates as [...]

]]> By: ParAccel actually uses relatively little PostgreSQL code | DBMS2 -- DataBase Management System Services http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-105843 ParAccel actually uses relatively little PostgreSQL code | DBMS2 -- DataBase Management System Services Tue, 30 Dec 2008 00:07:42 +0000 http://www.dbms2.com/?p=634#comment-105843 [...] I did get careless when I neglected to doublecheck something I already knew. The conclusion of this post isn’t really consistent with what ParAccel told me way back in February, 2007 about how much [...] [...] I did get careless when I neglected to doublecheck something I already knew. The conclusion of this post isn’t really consistent with what ParAccel told me way back in February, 2007 about how much [...]

]]> By: Omer Trajman http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-103951 Omer Trajman Wed, 10 Dec 2008 20:05:23 +0000 http://www.dbms2.com/?p=634#comment-103951 These are great questions and many are answered in the full benchmark report (http://www.vertica.com/php/pdfgateway?file=ETL-World-Record-Audit-Report.pdf). I can clarify trickle vs. bulk loads in Vertica and in this benchmark as well as what Vertica handles regarding real-time and near real time. The Vertica engine does not internally distinguish between trickle and bulk loads. There are no indexes to turn on and off, no transaction log and no special bulk load modes. Data is available for fast querying right after it commits. We do differentiate between trickle and bulk loads using common industry terminology. A trickle feed is where data is sent to the database at a rate that accommodates queries running concurrently and is typically continuous (24x7). A bulk load assumes no queries for some period of time and full resources are consumed by the data load. This benchmark was designed to test bulk loads – sending data to the server as fast as possible. At the end of the load data was immediately available for fast querying. We have numerous benchmarks (http://www.vertica.com/benchmarks) and customers (http://www.vertica.com/customers) that demonstrate trickle load capabilities. In telecommunications and financial services, trickle loads are a critical feature that prompted customers to deploy Vertica. Note that Vertica by itself is designed for near-real time (delay in seconds) not real-time or CEP analysis. We recently announced the deployment of Streambase and Vertica at BlueCrest Capital to handle both streaming data and historical data (http://www.vertica.com/company/news_and_events/StreamBase-and-Vertica-Announce-Customer-Deployment-at-BlueCrest). These are great questions and many are answered in the full benchmark report (http://www.vertica.com/php/pdfgateway?file=ETL-World-Record-Audit-Report.pdf). I can clarify trickle vs. bulk loads in Vertica and in this benchmark as well as what Vertica handles regarding real-time and near real time.

The Vertica engine does not internally distinguish between trickle and bulk loads. There are no indexes to turn on and off, no transaction log and no special bulk load modes. Data is available for fast querying right after it commits.

We do differentiate between trickle and bulk loads using common industry terminology. A trickle feed is where data is sent to the database at a rate that accommodates queries running concurrently and is typically continuous (24×7). A bulk load assumes no queries for some period of time and full resources are consumed by the data load.

This benchmark was designed to test bulk loads – sending data to the server as fast as possible. At the end of the load data was immediately available for fast querying. We have numerous benchmarks (http://www.vertica.com/benchmarks) and customers (http://www.vertica.com/customers) that demonstrate trickle load capabilities. In telecommunications and financial services, trickle loads are a critical feature that prompted customers to deploy Vertica.

Note that Vertica by itself is designed for near-real time (delay in seconds) not real-time or CEP analysis. We recently announced the deployment of Streambase and Vertica at BlueCrest Capital to handle both streaming data and historical data (http://www.vertica.com/company/news_and_events/StreamBase-and-Vertica-Announce-Customer-Deployment-at-BlueCrest).

]]> By: Jeremy Wong http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-103390 Jeremy Wong Thu, 04 Dec 2008 06:14:15 +0000 http://www.dbms2.com/?p=634#comment-103390 ETL is the legacy world, extract a whole thing and then load. Not really scalable. The active data warehouse is the future, when only incremental transactional changes to be loaded into data warehouse. See the DataMirror (now IBM), Shareplex or Wisdomforce that successfully do that for years ETL is the legacy world, extract a whole thing and then load. Not really scalable.
The active data warehouse is the future, when only incremental transactional changes to be loaded into data warehouse. See the DataMirror (now IBM), Shareplex or Wisdomforce that successfully do that for years

]]> By: Curt Monash http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-103356 Curt Monash Wed, 03 Dec 2008 19:02:47 +0000 http://www.dbms2.com/?p=634#comment-103356 Yep. Definitely ETL. Even mentioned that a small fraction of the records were deliberately erroneous. Yep. Definitely ETL. Even mentioned that a small fraction of the records were deliberately erroneous.

]]> By: Neil Raden http://www.dbms2.com/2008/12/02/data-warehouse-load-speeds-in-the-spotlight/#comment-103353 Neil Raden Wed, 03 Dec 2008 17:58:25 +0000 http://www.dbms2.com/?p=634#comment-103353 Seth, I haven't read the Vertica announcement yet, but my sense is that they were including ETL functions from Syncsort, not just screaming a neat flat file. I didn't see similar ETL comments in the other announcements. We also used to use sort utilities, including syncsort, to stage the files to improve bulk loading figures. So you have to look very carefully at these benchmarks. -NR twitter: nraden Seth, I haven’t read the Vertica announcement yet, but my sense is that they were including ETL functions from Syncsort, not just screaming a neat flat file. I didn’t see similar ETL comments in the other announcements.

We also used to use sort utilities, including syncsort, to stage the files to improve bulk loading figures. So you have to look very carefully at these benchmarks.

-NR
twitter: nraden

]]>