Curt,

That isn’t going to cut it. You can tell us old fuddy-duddy Oracle guys get peaved when a claim such as “Oracle took longer than 24 hours” and DATAllegro and Neteeza both took 2-3 minutes and you come back with some claims about how they messaged the data for queries. Let’s stick to the point.

The claim is that Oracle couldn’t ingest 100 million call detail records (CDR) in 24 hours. Even if they tested Oracle on a laptop and were limited to an ingest rate of 1MB/sec that would be
84GB with of 24hr load bandwidth…and, by the way, budget for 100 million CDRs of 900 bytes–which would be a HUGE CDR.

Please, for you reader’s sake, let’s see a little more intellectual curiosity go into these claims. If you think Oracle on any platform is limited to a bulk data load rate less than 1MB/s please, by all means come out and say it and then tell us what that has to do with an Oracle to Netezza/DATAllegro comparion.

If you can get something done in 1/700th the time, you’re either not doing the same thing (and I mean an apples to mucous comparison) or using 700 fold the resources. It’s as simple as that.

When it comes to getting records into a database elegance can only get you so much and much, much less than 700 fold. If not elegance, than brute force which can get 700 fold performance increase, but do us a favor and cite the configuration detail.

OK, I’m back to my work performance tuning a CDR loading exercise on my Pentium II laptop running Oracle Lite on Knoppix.

Curt,

I couldn’t agree with you more about balancing the pipes from I/O->CPU. And if you read as much of my blog as I yours, you’d know my position about trying to feed a huge hungry compute tier–be it a huge monolithic SMP or 50 RAC nodes or whatever–with little I/O tiny pipes.

So, Curt, when it comes down to it I suspect you are more biased toward BALANCED systems configurations than shared nothing or MPP…and it is for that reason the fangs remain concealed.

I agree that the TEOCO figure makes no sense on the surface. And indeed in the interests of time I often pass through claims without carefully vetting each one, but rather being careful about how I attribute them and the evidence (such as it is) for them. That said, the actual discussion suggested there were some extreme acts going on to massage the data to make it rapidly queryable later.

CAM

I apologize in advance for asking, but I have not found any “hardcore technical arguments” on why Oracle can not run well at these scales on your blog. Can you provide a few specific links?

I do realize this is a database analyst blog and not a database engineering blog, but some of your customer examples do not really seem well researched to me.

Let me give you one example of such a report:

“after extensive engineering…Oracle couldn’t get the load time for 100 million call detail records (CDRs) below 24 hours…DATAllegro and Netezza both handled it in 2-3 minutes.”

I find it completely bizarre that it does not seem that you even question why there is over a 480x-720x difference in load times (over 1440 mins vs. 2-3 mins). Do you think that DATAllegro/Netezza do something so unique and so revolutionary in loading data that it warrants a 480x-720x difference w/o even wondering why? I don’t have a Ph.D. in Mathematics, but something certainly does not add up to me. I for one think this has absolutely nothing to do with the database software. Do you agree, or is this simply just a case of: “This is what I was told and it fits my position on Oracle so I print it“?

I apologize again for the error re your posting date. Thanks for taking it so calmly.

As for whether MPP equates to shared-nothing — in theory, of course it doesn’t. But in practice the benefit of MPP is going to be lost unless you have a whole lot of channels communicating data from disks to processors at once. And so far, that equates to shared-nothing.

The problem is even more acute when you’re talking about RAM caches, and the speed of light comes seriously into play. To quote an old T-shirt, 3 x 10^10 cm/sec isn’t just a good idea. It’s the law.

Best regards,

CAM

I am a huge fan of MPP…especially with shared disk architecture. The two are not mutually exclusive.

I’ve been very short on blogging time lately. This should have turned into a post or two over at my blog.

It’s not really fair to say to someone that they “are dismissing a large fraction of the posts…” – I understand your point, but you’re not really expecting everyone who reads each post you’ve written, to then go and read a large proportion of your other posts to get all your supporting evidence, are you?

We’ve debated, briefly, this “Oracle doesn’t work with 5Tb+” issue a few months back and I don’t think we came to any significant conclusions then – it’s one of those debates that rumbles on I guess, with many differing viewpoints, however, I certainly didn’t leave the debate thinking I’d need to recommend to my client that they need to plan for a move to MPP.

As I said in the comments on that post we discussed a while back, I’ve no knowledge of MPP systems per se, but I just don’t understand how they offer more than a RAC or SMP solution that is architected properly – and yes, Greg is absolutely right, that architecting the thing right is the key – it’s the design principles that are more important here…not so much the hardware architecture, or software vendor for that matter. The individuals involved in the warehouse I work on are used to thinking about things in a warehouse way, rather than an OLTP way… perhaps that’s why it works for us.

If you fail to use appropriate database features such as partitioning, for example, then you should probably prepare to fail – I mentioned before that Tim Gorman wrote an excellent paper on this.

As Greg says, compression and parallelism are also very effective features to improve the chances of success. Bitmap indexes and materialized views can also aid performance too.

The data warehouse I deal with is currently 4.7Tb of real data – not indexes, temp or staging – I’m quoting it properly this time It’s likely to exceed 5Tb very shortly, and we’re still not planning on changing from Oracle any time soon.

Cheers
Jeff

Re: “If you have some technical evidence and details to support otherwise, do share.”

In writing that, you’re dismissing a large fraction of the posts in this blog as if they didn’t exist at all, from customer examples to hardcore technical arguments.

I really don’t have a guess as to what else I could say that you would regard as suitable.

CAM

I’m quite certain that your generalizations about Oracle data warehouses are related to the management of the database and not any limitations of the Oracle database software. If you have some technical evidence and details to support otherwise, do share.

When a data warehouse gets to 5TB – 10TB and one manages it like an OLTP database, it will certainly be painful. Then again, applying the same OLTP database management principles to any other data warehouse database software would result in the same pain. The other notable is the same principles that make and MPP DW perform well, also make an Oracle DW perform well. Specifically: appropriately choice of partitioning key and type, leveraging parallelism and using compression. The other common problem that hinders many Oracle data warehouses is inappropriate storage bandwidth provisioning. If sized appropriately for the workload, it makes a world of difference. Again, if one manages their Oracle data warehouse storage as if it was an OLTP database it will certainly be painful. If they follow the MPP model: adding storage, I/O bandwidth and CPU in building block increments, it again performs well. The issue that I see is that many Oracle shops brought in Oracle years ago as on OLTP database, gained experience with it in that form, then one day they built a data warehouse. This Oracle data warehouse was just another Oracle database so the same DBAs managed it in the same way they did all their other databases. This Oracle DW started small, then grew, and today it is slow, not because of any software limitations, but because the fundamental principles that make DW database software work well we not applied, or applied inappropriately.

There are also shops out there that thought that an MPP DW database (say like Teradata) would give them perfect scalability (as the marketing literature says) and they could just add more hardware as their database grew and keep the same performance. The only problem is that ratio between hardware and performance keeps growing and it gets more and more costly to keep the same performance as the DW grows. How could this be? Well, if good partition keys are not chosen, it results in much of the data being shipped between nodes. I’ve seen MPP DWs come to a near halt because of this. Again, if the fundamental principles of data warehousing are not applied, performance will suffer. Period. BTW, this also applies to the new data warehouse startups.