More from Vertica on data warehouse load speeds
Last month, when Vertica releases its “benchmark” of data warehouse load speeds, I didn’t realize it had previously released some actual customer-experience load rates as well. In a July, 2008 white paper that seems thankfully free of any registration requirements, Vertica cited four examples:
- (Comcast) Trickle loads 48MB/minute – SNMP data generated by devices in the Comcast cable network is trickle loaded on a 24×7 basis at rates as high as 135,000 rows/second. The system runs on 5 HP ProLiant DL 380 servers.
- (Verizon) Bulk loads to memory 300MB/minute – 50MB to 300MB of call detail records (1K record size—150 columns per row) are loaded every 10 minutes. Vertica runs on 6 HP ProLiant DL380 servers.
- (Level 3 Communications) Bulk loads to disk 5GB/minute – The loading and enrichment (i.e., summary table creation) of 1.5TB of call detail records formerly took 5 days in a row-oriented data warehouse database. Vertica required 5 hours to load the same data.
- (“Global investment firm”) Trickle loads 2.6GB/minute – Historic financial trade and quote (TaQ) data was bulk loaded into the database at a rate of 125GB/hour. New TaQ data was trickled into the database at rates as high as 90,000 rows per second (480b per row).
By the way, nobody should be surprised that when Vertica cites four production customers, three are in telecom and one in finance. Nor is it surprising that at least one of the four — Verizon — is also a big customer for Teradata and Netezza.
The first point that pops out is that the two figures for “trickle loads” differ by >50X. But the larger figure is actually fewer rows per minute, give or take the weasel words “as high as.” Presumably the number of columns in the two databases isn’t dissimilar.
Beyond that, these figures highlight that load-speed considerations vary greatly from database to database. And, oh yeah, Vertica is fast enough to handle data warehouse load requirements in some pretty demanding situations.
Comments
Leave a Reply