High-performance analytics

For the past few months, I’ve collected a lot of data points to the effect that high-performance analytics – i.e., beyond straightforward query — is becoming increasingly important. And I’ve written about some of them at length. For example:

• MapReduce – controversial or in some cases even disappointing though it may be – has a lot of use cases.
• It’s early days, but Netezza and Teradata (and others) are beefing up their geospatial analytic capabilities.
• Memory-centric analytics is in the spotlight.

Ack. I can’t decide whether “analytics” should be a singular or plural noun. Thoughts?

Another area that’s come up which I haven‘t blogged about so much is data mining in the database. Data mining accounts for a large part of data warehouse use. The traditional way to do data mining is to extract data from the database and dump it into SAS. But there are problems with this scenario, including:

Read more

Big scientific databases need to be stored somehow

A year ago, Mike Stonebraker observed that conventional DBMS don’t necessarily do a great job on scientific data, and further pointed out that different kinds of science might call for different data access methods. Even so, some of the largest databases around are scientific ones, and they have to be managed somehow. For example:

• Microsoft just put out an overwrought press release. The substance seems to be that Pan-STARRS — a Jim Gray legacy also discussed in an August, 2008 Computerworld article — is adding 1.4 terabytes of image data per night, and one not so new database adds 15 terabytes per year of some kind of computer simulation output used to analyze protein folding. Both run on SQL Server, of course.
• Kognitio has an astronomical database too, at Cambridge University, adding 1/2 a terabyte of data per night.
• Oracle is used for a McGill University proteonomics database called CellMapBase. A figure of 50 terabytes of “mass storage” is included, which doesn’t include tape backup and so on.
• The Large Hadron Collider, once it actually starts functioning, is projected to generate 15 petabytes of data annually, which will be initially stored on tape and then distributed to various computing centers around the world.
• Netezza is proud of its ability to serve images and the like quickly, although off the top of my head I’m not thinking of a major customer it has in that area. (But then, if you just sell software, your academic discount can approach 100%; but if like Netezza you have an actual cost of goods sold, that’s not as appealing an option.)

Long-term, I imagine that the most suitable DBMS for these purposes will be MPP systems with strong datatype extensibility — e.g., DB2, PostgreSQL-based Greenplum, PostgreSQL-based Aster nCluster, or maybe Oracle.

How to tell Teradata’s product lines apart

Once Netezza hit the market, Teradata had a classic “disruptive” price problem – it offered a high end product, at a high price, sporting lots of features that not all customers needed or were willing to pay for. Teradata has at times slashed prices in competitive situations, but there are obvious risks to that, especially when a customer already has a number of other Teradata systems for which it paid closer to full price.

This year, Teradata has introduced a range of products that flesh out its competitive lineup. There now are three mainstream Teradata offerings, plus two with more specialized applicability. Teradata no longer has to sell Cadillacs to customers on Corolla budgets.

But how do we tell the five Teradata product lines apart? The names are confusing, both in their hardware-vendor product numbers and their data-warehousing-dogma product names, especially since in real life Teradata products’ capabilities overlap. Indeed, Teradata executives freely admit that the Teradata Data Mart Appliance 551 can run smaller data warehouses, while the Teradata Data Warehouse Appliance 2550 is positioned in large part at what Teradata quite reasonably calls data marts.

When one looks past the difficulties of naming, Teradata’s product lineup begins to make more sense. Let’s start by considering the three main Teradata products.

Read more

Eric Lai on Oracle Exadata, and some addenda

Eric Lai offers a detailed FAQ on Oracle Exadata, including a good selection of links and quotes. I’d like to offer a few comments in response: Read more

Netezza and Teradata on analytic geospatial data management

Geospatial data management is one of the flavors of the month:

• Last week, Teradata claimed it has the most sophisticated analytic geospatial data management capability.
• Also last week, Netezza’s newly acquired Netezza Spatial technology attracted a lot of attention.
• This week, Oracle called attention to its geospatial capabilities.

So I asked Netezza and Teradata what this geospatial analytics stuff is all about.

Read more

So what does Oracle Exadata mean for HP Neoview?

That HP is committed to selling a lot of data warehouse hardware — and probably data warehouse appliances in particular — seems obvious, for reasons including:

• HP bought a big BI/data warehousing consulting operation in Knightsbridge.
• HP has put considerable effort into its data warehouse appliance Neoview.
• HP CEO Mark Hurd comes from data warehouse appliance vendor Teradata.
• Data warehousing where the big bucks are.

But Oracle Exadata could produce those appliance sales. So where does HP Neoview fit in?

I was told by an investor today that HP’s investor relations department is saying Oracle Exadata is a Netezza competitor, while Neoview is more in the Teradata market. That’s laughable. Read more

Peter Batty on Netezza Spatial

As previously noted, I’m not up to speed on Netezza Spatial. Phil Francisco of Netezza has promised we’ll fix that ASAP. In the mean time, I found a blog by a guy named Peter Batty, who evidently:

• Knows a lot about geospatial data and its uses
• Is consulting to Netezza
• Is smart

Batty offers a lot of detail in two recent posts, intermixed with some gollygeewhiz about Netezza in general. If you’re interested in this stuff, Batty’s blog is well worth checking out. Read more

Web analytics — clickstream and network event data

It should surprise nobody that web analytics – and specifically clickstream data — is one of the biggest areas for high-end data warehousing. For example:

• I believe that both of the previously mentioned petabyte+ databases on Greenplum will feature clickstream data.
• Aster Data’s largest disclosed database, by almost two orders of magnitude, is at MySpace.
• Clickstream analytics is a big application area for Vertica Systems.
• Clickstream analytics is a big application area for Netezza.
• Infobright’s customer success stories appear to be concentrated in clickstream analytics.
• Coral8 tells me that CEP is also being used for clickstream data, although I suspect that a lot of Coral8’s evidence in that regard comes from a single flagship account. Edit: Actually, Coral8 has a bunch of clickstream customers.

Read more

Netezza overseas

22% of Netezza’s revenue comes from outside the US, at least if we use last quarter’s figures as a guide.  At first blush, that doesn’t sound like much.  Indeed, percentage-wise it surely lags behind Teradata, Greenplum (which has sold a lot in Asia/Pacific under Netezza’s former head of that region), and a few smaller competitors headquartered outside the US.  But a few conversations I had today suggest a rosier view.  Read more

Netezza application areas

I’m at the Netezza “Enzee” user conference in Orlando.  So one or more Netezza posts are in order.

One theme of the brief analyst meeting was Netezza’s increasing business focus on vertical markets.  In particular, Netezza is hiring managers for a range of vertical markets.  The commercial ones cited (at various levels of maturity) included: Read more

More mysteries regarding Oracle CDR load speed

Last spring, DATAllegro user John Devolites of TEOCO told me of troubles his firm had had loading CDRs (Call Detail Records) into Oracle, and how those had been instrumental in his eventual adoption of DATAllegro.  That claim was contemptously challenged in a couple of comment threads.

Well, tonight at the Netezza user conference, Netezza gave awards to its first customers.  The very first to accept was Jim Hayden, who’d bought Netezza for a company called Vibrant Solutions, which coincidentally was later acquired by TEOCO itself.  In front of hundreds of people, he talked about how, back in 2003, it had taken 23 hours to load 400 million CDRs into Oracle on Nextel’s behalf, but only 40 minutes on Netezza.

And I’ll erase the rest of what I’d drafted here, as it was dripping in sarcasm …

Teradata decides to compete head-on as a data warehouse appliance vendor

In a press release today that is surely timed to impinge on the Netezza user conference news cycle, Teradata has come out swinging. Highlights include:

• Teradata, which long avoided the “appliance” term, now says it sells both “data warehouse appliances” and “data mart appliances.” Indeed, it claims to have “invented the original appliance” — which is pretty close to being true.*
• Teradata claims its “new appliance easily delivers up to 5 to 10 times performance improvement over competitors’ appliances,” at $119,000 per terabyte US list price.
• Teradata claims a 150% faster “scan rate” than competitors. Teradata is surely thinking of Netezza when saying that.
• Teradata claims 10X performance improvement on “selected queries” vs. the “competition.”
• Teradata thinks its geospatial data management capability is better than competitors’, and that this is an important indicator of Teradata’s general overall greater sophistication.

Read more

Some Netezza customer metrics

From the conference call based on Netezza’s July, 2008 Q1, as of the end of Q1:

• There are now 191 Netezza customers.
• 18 of those were new.
• 78% of Netezza’s business was in North America and 22% was international.
• Netezza operates in 10 countries.
• “The top 4 vertical markets represented approximately 75% of our business, with those markets being telcos, retail, financial services, and the analytic service provider segment. “
• One analytic service provider was greater than 10% of revenue for the quarter, and is expected to keep buying a lot in subsequent quarters. Also, one analytic service provider standardized on Netezza. I’m guessing that’s the same customer.
• “We ended the quarter with 45 [quota] carrying teams made up of a sales rep and a systems engineer and our plan is to continue to hire direct sales teams at the pace of 3 to 5 per quarter every quarter. These direct reps accounted for 85% of the business while the indirect activity was 15% this quarter.”

Teradata/Netezza/Tesco kerfuffle

Netezza evidently put out a press release bragging of a competitive replacement of Teradata at UK retailing giant Tesco. That press release cannot be now found on Netezza’s site, but it lives on elsewhere. Meanwhile, Teradata has put out a press release in which Tesco is quoted emphatically contradicting what it is quoted as saying in the Netezza press release. While I haven’t discussed this with Netezza, my guess is that somebody there got a little overenthusiastic in advance of their user conference next week and thought they’d gotten a permission they really hadn’t.

Beyond that, I’d note that the Netezza quote made reference to around 25 heavy analytical users, while the Teradata quote talked of 8000 people across more than 2000 suppliers.

Read more

The layered messaging marketing model as applied to Netezza

I just put up a post claiming that enterprise IT marketing arguments commonly boil down to one of two layered messaging templates. Let’s test how that claim applies to one of the most innovative technology companies of this decade: Netezza.

Read more

More data on data warehouse sizes and issues

I spoke today with Paul Barth and Randy Bean of consultancy NewVantage Partners. The core of NewVantage’s business seems to be helping large enterprises (especially financial services) with their data warehouse strategies. Takeaways — none of which should shock regular readers of DBMS2 — included:

• Administrative cost and difficulty are often the single biggest issue in selecting analytic DBMS products.
• Oracle hits a wall around 10 terabytes of user data. The one customer NewVantage can think of with an Oracle data warehouse over 10 terabytes is fleeing Oracle for Netezza.
• NewVantage says that very specialized data warehouses on Oracle could conceivably be larger than that.
• NewVantage does have a customer on DB2/UDB in the 30-40 terabyte range. That customer does a lot of careful tuning to make it work.
• About 15% of NewVantage’s customers use Netezza. Few if any use newer analytic DBMS (but I got the sense more will soon). The rest rely on “traditional” DBMS, a group that includes Teradata.

Enterprises are buying multiple brands of analytic DBMS each

Over the past few weeks I’ve had a lot of NDA discussions about analytic DBMS vendors’ specific customers. And so I’ve been acutely aware of something I already sort of knew — just as there was in prior generations of database management technology, there’s huge overlap among analytic DBMS vendors’ customer bases as well. As they always have, enterprises are investing in multiple different brands of DBMS, even in cases where those DBMS can do pretty much the same things.

For example:

• Many Teradata users are buying newer technology too. But they aren’t actually throwing out Teradata.
• The same sometimes applies to Netezza already. At least two Netezza references are also references for a rival vendor.
• One outfit is among the biggest customers for two different analytic DBMS vendors, neither of which is Teradata or Netezza.
• One corporation is using or deploying four different brands of analytic DBMS.
• TEOCO is a big user of both DATAllegro and Netezza.

My current customer list among the data warehouse specialists

One of my favorite pages on the Monash Research website is the list of many current and a few notable past customers. (Another favorite page is the one for testimonials.) For a variety of reasons, I won’t undertake to be more precise about my current customer list than that. But I don’t think it would hurt anything to list the data warehouse DBMS/appliance specialists in the group. They are:

• Aster Data
• Calpont
• DATAllegro
• Greenplum
• Infobright
• Netezza
• ParAccel
• Teradata
• Vertica

All of those are Monash Advantage members.

If you care about all this, you may also be interested in the rest of my standards and disclosures.

Netezza update

In my usual dual role, I called Phil Francisco of Netezza to lay some post-Microsoft/DATAllegro consulting on him late on a Friday night — and then took the opportunity of being on the phone with him to get a general Netezza update. Netezza’s July quarter just ended, so they’re still in quiet period, so I didn’t press him for a lot of numerical detail. More generally, I didn’t find a lot out that wasn’t already covered in my May Netezza update. But notwithstanding all those disclaimers, it was still a pretty interesting chat. Read more

Data warehouse appliance power user TEOCO

If you had to name super-high-end users of data warehouse technology, your list might start with a few retailers, credit data processors, and telcos, plus the US intelligence establishment. Well, it turns out that TEOCO runs outsourced data warehouses for several of the top US telcos, making it one of the top data warehouse technology users around.

A few weeks ago, I had a fascinating chat with John Devolites of TEOCO. Highlights included:

• TEOCO runs a >200 TB DATAllegro warehouse for a major US telco. (When we hear about a big DATAllegro telco site that’s been in production for a while, that’s surely the one they’re talking about.)
• TEOCO runs around 450 TB total of DATAllegro databases across its various customers. (When Stuart Frost blogs of >400 TB “systems,” that may be what he’s talking about.)
• TEOCO likes DATAllegro better than Netezza, although the margin is now small. This is mainly for financial reasons, specifically price-per-terabyte. When TEOCO spends its own money without customer direction as to appliance brand, it buys DATAllegro.
• TEOCO runs at least one 50 TB Netezza system — originally due to an acquisition of a Netezza user — with more coming. There also is more DATAllegro coming.
• TEOCO feels 15-30 concurrent users is the current practical limit for both DATAllegro and Netezza. That’s greater than it used to be.
• Netezza is a little faster than DATAllegro on a few esoteric queries, but the difference is not important to TEOCO’s business.
• Official price lists notwithstanding, TEOCO sees prices as being in the $10K/TB range. DATAllegro’s price advantage has shrunk greatly, as others have come down to more or less match. However, since John stated his price preference for DATAllegro as being in the present tense, I presume the price match isn’t perfect.
• Teradata was never a serious consideration, for price reasons.
• In the original POC a few years ago, the incumbent Oracle — even after extensive engineering — couldn’t get an important query down under 8 hours of running time. DATAllegro and Netezza both handled it in 2-3 minutes. Similarly, Oracle couldn’t get the load time for 100 million call detail records (CDRs) below 24 hours.
• Applications sound pretty standard for telecom: Lots of CDR processing — 550 million/day on the big DATAllegro system cited above. Pricing and fraud checking. Some data staging for legal reasons (giving the NSA what it subpoenas and no more).

Netezza on compression

Phil Francisco put up a nice post on Netezza’s company blog about a month ago, explaining the Netezza compression story. Highlights include:

• Like other row-based vendors, Netezza compresses data on a column-by-column basis, then stores the results in rows. This is obviously something of a limitation — no run-length encoding for them — but can surely accommodate several major compression techniques.
• The Netezza “Compress Engine” compresses data on a block-by-block basis. This is a disadvantage for row-based systems vs. columnar ones in the area of compression, because columnar systems have more values per block to play with, and that yields higher degrees of compression. And among row-based systems, typical block size is an indicator of compression success. Thus, DATAllegro probably does a little better at compression than Netezza, and Netezza does a lot better at compression than Teradata.
• Netezza calls its compression “compilation.” The blog post doesn’t make the reason clear. And the one reason I can recall confuses me. Netezza once said the compression extends at least somewhat to columns with calculated values. But that seems odd, as Netezza only has a very limited capability for materialized views.
• Netezza pays the processing cost of compression in the FPGA, not the microprocessor. And so Netezza spins the overhead of the Compress Engine as being zero or free. That’s actually not ridiculous, since Netezza seems to have still-unused real estate on the FPGA for new features like compression.

Read more

Netezza has an EMC deal too

Netezza has an EMC deal too. As befits a hardware vendor, Netezza has an actual OEM relationship with EMC, in which it is offering CLARiiONs built straight into NPS appliances. 5 TB of CLARiiON will be free in any Netezza system from 2 racks on upward. (A rack holds about 12.5 TB.) In addition, you’ll be able to buy 10 TB more of CLARiiON in every Netezza rack, if you want. The whole thing is supposed to ship before year-end. Read more

Netezza, enterprise data warehouses, and the 100 terabyte mark

Phil Francisco of Netezza checked in tonight with some news that will be embargoed for a few hours. While I had him on the phone anyway, I asked him about large databases and/or enterprise data warehouses. Highlights included:

• Netezza has one customer with 200 TB of user data. The name is confidential (but he told me who it was).
• Netezza has sold 15 or so of its NPS 10-800s, which are rated at 100 TB capacity.
• The second-largest database in production on Netezza is probably 80 TB or so at Catalina Marketing, which has been a Netezza early adopter all along.
• Netezza’s biggest users typically have a handful (literally — off the top of his head, Phil said “4 to 6″) of applications, each with its own primary set of fact tables.
• Each application-specific set of fact tables in such big-honking-data-mart installations is usually either of cardinality one, or else a small set sharing a common hash key.
• Phil insists Netezza isn’t exaggerating when it claims to have true enterprise data warehouse installations. What he means by an EDW is something that is an enterprise’s primary data warehouse, is used by lots of departments, draws data from lots of sources, has loads going on at various points during the day, and has 100s if not 1000s of total users.
• Netezza’s biggest EDW has about 30 TB of user data. Phil wouldn’t tell me the name of that customer.

Outsourced data marts

Call me slow on the uptake if you like, but it’s finally dawned on me that outsourced data marts are a nontrivial segment of the analytics business. For example:

• I was just briefed by Vertica, and got the impression that data mart outsourcers may be Vertica’s #3 vertical market, after financial services and telecom. Certainly it seems like they are Vertica’s #3 market if you bundle together data mart outsourcers and more conventional OEMs.
• When Netezza started out, a bunch of its early customers were credit data-based analytics outsourcers like Acxiom.
• After nagging DATAllegro for a production reference, I finally got a good one — TEOCO. TEOCO specializes in figuring out whether inter-carrier telcom bills are correct. While there’s certainly a transactional invoice-processing aspect to this, the business seems to hinge mainly around doing calculations to figure out correct charges.
• I was talking with Pervasive about Pervasive Datarush, a beta product that lets you do super-fast analytics on data even if you never load it into a DBMS in the first place. I challenged them for use cases. One user turns out to be an insurance claims rule-checking outsourcer.
• One of Infobright’s references is a French CRM analytics outsourcer, 1024 Degres.
• 1010data has built up a client base of 50-60, including a number of financial and retail blue-chippers, with a soup-to-nuts BI/analysis/columnar database stack.
• I haven’t heard much about Verix in a while, but their niche was combining internal sales figures with external point-of-sale/prescription data to assess retail (especially pharma) microtrends.

To a first approximation, here’s what I think is going on. Read more

Netezza pricing

In connection with the announcement of the Teradata 2500, I asked some Teradata competitors about pricing. Netezza’s response amounted to “We don’t disclose list pricing, but our cheapest system handles about 3 1/4 TB and sells for under $200K.” So Netezza’s actual pricing is well below the list price of the Teradata 2500.

Next Page →

Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

• DBMS2 covers database management, analytics, and related technologies.
• Text Technologies covers text mining, search, and social software.
• Strategic Messaging analyzes marketing and messaging strategy.
• The Monash Report examines technology and public policy issues.
• Software Memories recounts the history of the software industry.

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Recent white paper

The Explosion in DBMS Choice August, 2008

Recent webcast

What leading database vendors don't want you to know Originally broadcast April 9, 2008

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.

 

• Recent posts

  • Interpreting the results of data warehouse proofs-of-concept (POCs)
  • MySQL Query Analyzer
  • Silly website tricks
  • Graphjam: I can haz BI
  • When people don’t want accurate predictions made about them

• Categories

  • About this blog
  • Analytic technologies
    • Business intelligence
    • Data mart outsourcing
    • Data warehousing
    • MOLAP
  • Application areas
    • Games and virtual worlds
    • Investment research and trading
    • Scientific research
    • Telecommunications
    • Web analytics
  • Companies and products
    • 1010data
    • Ab Initio Software
    • Alpha Five
    • Amazon and its cloud
    • ANTs Software
    • Aster Data
    • Business Objects
    • Calpont
    • Cast Iron Systems
    • Cogito and 7 Degrees
    • Cognos
    • Coral8
    • CouchDB
    • DATAllegro
    • Dataupia
    • Elastra
    • EMC
    • EnterpriseDB and Postgres Plus
    • Exasol
    • Facebook and Cassandra
    • FileMaker
    • Google
    • Greenplum
    • H-Store
    • HP and Neoview
    • IBM and DB2
      • pureXML
    • illuminate Solutions
    • Infobright
    • Informatica
    • Ingres
    • Intel
    • Intersystems and Cache'
    • Jaspersoft
    • Kalido
    • Kickfire
    • Kognitio
    • Mark Logic
    • McObject
    • Microsoft and SQL*Server
    • MySQL
    • Netezza
    • Oracle
      • Exadata
    • Oracle TimesTen
    • ParAccel
    • Pervasive Software
    • PostgreSQL
    • Progress, Apama, and DataDirect
    • QlikTech and QlikView
    • SAP AG
    • SAS Institute
    • ScaleDB
    • Software AG
    • solidDB
    • StreamBase
    • Sybase
    • Talend
    • Teradata
    • Truviso
    • Vertica Systems
  • Data integration and middleware
    • Application servers
    • EAI, EII, ETL, ELT, ETLT
  • Data types
    • GIS and geospatial
    • Native XML
    • Object
    • RDF and graphs
    • Text
  • DBMS product categories
    • Data warehouse appliances
    • Mid-range
    • OLTP
    • Open source
  • Emulation, transparency, portability
  • Fun stuff
    • Humor
  • Market share
  • Memory-centric data management
    • Cache
    • Complex event processing (CEP)
    • In-memory DBMS
  • Michael Stonebraker
  • Parallelization
    • MapReduce
  • Pricing
  • Software as a Service (SaaS)
    • Cloud computing
  • Specific users
    • eBay
    • TEOCO
    • Yahoo
  • Storage
    • Solid-state memory
  • Theory and architecture
    • Columnar database management
    • Data models and architecture
    • Database compression
    • Database diversity
  • TransRelational
  • Uncategorized

• Date archives

  Select Month November 2008  (10) October 2008  (29) September 2008  (60) August 2008  (28) July 2008  (23) June 2008  (7) May 2008  (16) April 2008  (18) March 2008  (22) February 2008  (25) January 2008  (35) December 2007  (18) November 2007  (12) October 2007  (31) September 2007  (16) August 2007  (11) July 2007  (9) June 2007  (16) May 2007  (6) April 2007  (18) March 2007  (17) February 2007  (8) January 2007  (13) December 2006  (1) November 2006  (1) October 2006  (7) September 2006  (17) August 2006  (6) July 2006  (12) June 2006  (1) May 2006  (7) April 2006  (7) March 2006  (2) February 2006  (4) January 2006  (13) December 2005  (9) November 2005  (9) October 2005  (10) September 2005  (1) August 2005  (6)

• Links

  • Monash Research
  • Webcasts
  • White Papers

• Admin

  • Log in

• Home
• About
• Contact
• Feeds