Looks like Greg got bored of waiting for the list of “tricks” and made his own post on it shortly after…great stuff!

Cheers
Jeff

http://structureddata.org/2008/04/28/top-ways-how-not-to-scale-your-data-warehouse/

You have mentioned one of the top issues with Oracle data warehouses: under-configured I/O bandwidth. The MPP architecture itself is not immune to this problem, it’s just that the vendors that use it dictate the hardware configuration, much in the same way Henry Ford dictated that every Model T was black. If Teradata (or any other MPP vendor) let its customers choose the storage and how it was connected to the hosts (# of HBAs), they would have exactly the same problems.

I also agree there is no “bag of tricks” needed in using Oracle for data warehousing, at least no more than there is with any other vendor. It’s all about having good design and applying the appropriate feature(s) to the problem(s). Like you, I’d be quite interested to see a list of “tricks” that is “1-2 dozen” long. My guess is these “tricks” are fundamentals of data warehouse design.

IO issues revolve around a number of things.

1. The system isn’t balanced…so, the SAN has ports which can deliver 1.6GBytes/Sec and the server has CPU capability to drive 3.2GBytes/Sec…but the number/type of HBAs on the back of the server can only handle 780MBytes/Sec so it’s the limiting factor and means all the other components are having an easy time of it. The SAN is also shared by other apps, uses RAID 5 (Yeah I know…don’t tell Mogens!) in 3d+1p format and uses veritas with quick io…there are a lot of layers in the IO stack

2. Somewhere there is/are some IO problems with even reaching this 780MBytes/Sec limit, because even on a clear machine (no other users to contend with), we still only manage to drive about 680MBtes/Sec via a big parallel full table scan of a large (80Gb) table…so something somewhere is wrong in the IO stack…and my point there would be, that the same IO stack is going to sit under whichever database platform you care to use…so if it’s not performing for one DBMS vendor, then it won’t matter if you switch the vendor…your IO still sucks.

Are the users getting all the data they want…No. Are they getting lots more than they were on their previous systems…Yes.

Why don’t they get want they want? Well, many reasons – some of which can be levelled at the performance / variability of performance and some of which are more about their education level (they use SQL and are not as effective with it as they need to be), application/data model design and contention with existing batch load…which if the IO stack was working better, the it would have finished before the users get online during the day.

It’s not about one simple thing for us…it’s many variables we’re juggling. I’ve heard they are actually going to buy some new kit (typical management response to performance issues – throw hardware at the problem)…hopefully that kit will be built around a balanced, high throughput IO stack…otherwise the problems which do exist for our system, will remain.

Why is an MPP system going to be better at doing the IO?

I’m no MPP expert – I’ve never worked on one as it’s not really an Oracle thing…you can get Oracle on MPP I believe. My understanding of MPP is that each node in the cluster will have it’s own disks and that means you need to manually partition your data across the nodes and onto the disks that are available to that node….so what’s the difference between that and a good partitioning strategy on the SMP environment with the shared disk subsystem? You need to lay out your stuff in an appropriate fashion to ensure that the IO is balanced across all the spindles available…which you’d do whether it was SMP or MPP.

As I said, I’m no MPP expert so I don’t get why MPP would help…if anything it’s harder because the kit is more difficult to administer? and you have to manually partition the data across the nodes?…which may or may not be easy/possible and may require more work over time?

Cheers
Jeff

That makes sense — but could you please say more about those “general” I/O problems? They may very well be exactly the kind of thing that MPP shared-nothing architectures are designed to circumvent.

Also — is it really the case that your users are getting all the data they want, as quickly as they want it?

Best,

CAM

I’ll openly admit, I’m an Oracle biased person…not because I’ve compared and contrasted various products, but purely because I’ve been working with Oracle technologies since the days of Oracle 5 and forms 2.0 and that’s longer than I’d care to remember! I’m sure the other vendors in the RDBMS space are all good at what they do…but by and large, I’ve generally been very happy with what Oracle has offered me to deliver projects over the years.

I probably get a bit annoyed and I’m a little too quick to jump off the deep end when I hear, what I perceive to be, anybody “dissing” Oracle…so forgive me if I appeared to be quick to get on the defensive.

With regard to using Oracle as the database engine for our warehouse, I still don’t feel that we will be changing that anytime soon…the problems we have are not inconsequential…but they are definitely not the database engine itself…more like general IO capabilities and system design/procedures really…both of which are big factors in getting a warehouse to work on any database engine.

It’s interesting, the boys who parachuted into this company and said “we need a warehouse” came from a big bank where they had access to a Teradata Warehouse so from day one we’ve had a battle as to the database engine to be used. I’m still of the opinion that until we prove Oracle isn’t capable, then we should leave it as is…they are an Oracle shop, with zero Teradata skills, so converting would be a costly and time consuming process and I’m just not convinced there would be any tangible benefits. They seem to have this fantasy that they can just “convert” it to Teradata by installing the software and then all of a sudden their performance and functionality issues will disappear.

Time will tell…but if it does go Teradata, I’m not likely to be around to see the results as it’s just not my area of expertise…mind you, I should check out the rates for a Teradata contractor before I say that!

Cheers
Jeff

Back in the 1990s I wrote my first vendor sponsored piece ever. Sybase, Ingres, et al. said “Why should we sponsor this? You love Oracle and hate us!” Oracle said “Why should we sponsor this? You love our competitors and hate us!” Thankfully, most of them sponsored anyway …

In the recent past, I’ve been criticized — sometimes as gently as you just did, sometimes more roughly — for, among other things, being:

Anti-Oracle
Anti-Teradata
Pro-Teradata
Anti-Netezza
Pro-Netezza
Anti-relational
Pro-relational
Anti-MySQL
Pro-MySQL

And that’s just off the top of my head.

By the way, in political discussions I am commonly criticized for being too liberal and too conservative. In my personal life I am criticized for being a rebel and a fuddy-duddy.

Where you stand depends upon where you sit. I’m used to it.

Anyhow, thanks for sharing those figures. You’re right up in the range where I think Oracle still does a perfectly decent job for lots of folks. But if you had to double the size of your warehouse in a year, would you truly feel comfortable about staying with Oracle? If you had to quadruple it, would you look actively for other alternatives?

Best,

CAM

I think you’re thinking in the right direction.

For many years, software vendors had relatively little in the way of economies of scale in software development, to an extent that would be surprising to anybody who hadn’t either worked in the area or, say, read “The Mythical Man-Month”.

But at a certain point the economies of scale became very real, less as a way to gain advantage than as a way to hold advantage gained in the kinds of ways that Geoffrey Moore made a career out of explaining.

The replacement buzz-theory for Crossing the Chasm is The Innovator’s Dilemma, and I think the high-end software vendors are running straight into that kind of disruption. That doesn’t mean they won’t win. They’re flexible enough to make acquisitions, and I think the economies of scale in selling SaaS to mid-range enterprises are still UNDER-appreciated. But I think the odds of “disruptive” TECHNOLOGIES winning is quite strong, even if ultimately that victory takes the form of a high-priced buyout by a market-leading vendor.

CAM

Fair do’s…I’m quoting database size from the Oracle Enterprise Manager front screen which currently shows 6.5Tb…but yes, that includes indexes and temp and scratch and other “non data”…raw data…about 4Tb I think…and growing at 1.5Tb (data) a year.

Hardware is a 32 way HP RP8420 box with 128Gb RAM and a HDS USP100 SAN for storage. The box is unstressed, although the same cannot be said of the IO subsystem.

System has five fact tables.

We do have some performance issues…but they are mainly at the IO level…our system is not well balanced and is not providing what the theoretical hardware limits suggest it should…so we’re investigating things like that…but **generally** the thing runs well, performs well and answers lots of new business questions that couldn’t be answered with the previous MI systems…it’s providing a ROI.

I think I took a little umbridge at your suggestion that things like Partitioning are “tricks”. It’s a feature, not a trick – if using a feature like this is a trick then I’m Penn and Teller! Anybody who tries to build any MI environment with time series data (e.g. a warehouse) needs their head reading if they choose to do it without partitioning – Tim Gorman wrote an excellent article about this called “Scaling to infinity”.

I don’t know you, but you sound like a pro Teradata person and an anti Oracle one. I know nothing about Teradata and I’m sure it’s good at what it does…but I’m also fairly sure that the biggest problems in setting up a warehouse are to do with how you architect it – hardware, OS, filesystem, logical database design etc…get those right and I don’t see why Oracle can’t succeed with 10Tb databases…I’m sure there is an element of marketing in it but the Winter Corporation results for 2005 (http://www.oracle.com/corporate/press/2005_sep/091305_wintertopten_finalsite.html) identify a 100Tb Oracle database back then…and we’ve moved on further since then somewhat.