I would agree with your interpretation of the dual/quad fibre channel [card]. To be honest, I would have never guessed anyone made a 4 port HBA but I guess LSI does: LSI7404XP-LC and given Teradata resells LSI Engenio storage it is likely they use their HBAs also. Given a 2 port 4GFC PCI-X HBA can deliver 80% of the slot bandwidth, it seems like a bit of a waste to go to 4 ports, at least for performance. For connectivity, perhaps, which is why I believe Teradata may use it – for their cliques.

The other reason for my comment of a max of 1600MB/s per node is that the LSI Engenio 6998 array only does a max of 1600MB/s (per LSI’s presentation) and the 7900 array is quite new so it would seem doubtful that eBay uses that one. They may have opted for the EMC DMX storage, but I would think that would be an extremely costly solution at 72 nodes.

I may soften up a bit on the I/O rate also. Curt’s comment “maybe that’s a peak when the workload is scan-heavy” is probably correct. Peak rate vs. sustained, I could see that. Maybe they do some light scans of some large de-normalized table making it peak out at 2GB/s per node. But that number still seems quite high at 250MB/s per CPU core. Doing group bys and aggregation I’m sure that number drops fast.

I think the interesting, and unmentioned data point, is how many hard drives are in this 72 node config to deliver this I/O number. eBays’s own Michael McIntire reports that Teradata I/O is all random so the MBPS rate per drive is probably somewhere around 30MB/s (give or take). My guess is that there is somewhere between 4500 and 5000 HDDs (around 64 HDDs per node).

Without performance regards of end user queries, the size of DB is meaningless, at most is data storage. There are some limits of VLDB being set up by system considering both size and performance of DB. It would be nice to have some numbers for this.

Yan

I went to the link and looked up the specs on the Teradata 5550 node. The data sheet says there are 3 PCI-X slots. It also says I/O can be 4 GB dual or quad fibre channel. My interpretation is dual or quad PCI-X adapter cards. With 3 PCI-X slots that means Teradata can have up to 12 Fibre Channel links per node for a theoretical bandwidth of 4.8 GB/s. The limiting factor is probably three 133 MHz PCI-X buses, which are 1066 MB/s apiece giving 3 GB/s per node.

Mr Rahn also says “I would also comment that 2 Quad Core Xeon 5400 series processors would not be able to do anything but a SELECT COUNT(*) and ingest data at 2GB/s (or even 1600MB/s).” But yet eBay says they are doing it – and a lot more.