• Personally selected annual lists of 12 “Most influential” companies and 36 “Companies to watch” in analytics- and database-related sectors.
• Made it clear that these are his personal selections.
• Nonetheless has called it an Editors’ Choice list, rather than Editor’s Choice.

(Actually, he’s really called it an “award.”)

People advising Doug — who come to think of it actually are Contributing Editors to Intelligent Enterprise or something like that — included Cindi Howson, Seth Grimes, three others, and me.

And if past is prologue, I will now get a flood of PR emails calling my attention to this award that I already have both participated in and blogged about.

As usual, the sense:nonsense ratio on these lists was pleasingly high. Analytic DBMS vendors cited included IBM, Microsoft, Netezza, Oracle, Sybase, and Teradata in the “Most influential” group, with Aster, Greenplum, HP, Infobright, and Vertica among the “To watch” crowd. It’s tough to argue with those selections, whose most questionable element is probably the not-ridiculous supposition that HP could do something interesting over the coming year. Cloudera and Intersystems also made the list, deservedly.

All three of QlikTech, Tableau, and TIBCO made the list, which is appropriate given the potential for and interest in interactive data exploration technology.  The BI majors, independent or otherwise, were all on as well. In text mining, Doug included Attensity and Clarabridge, which I think is exactly right. (Plus OpenCalais.)  Upon reflection, I probably should have nominated Mark Logic, even though most of its business is non-enterprise; but hey, nobody’s perfect, and the same goes for lists. Open source was well represented, with Apache, Actuate, Jaspersoft, Eclipse, Infobright, Nuxeo and R all being cited (but not Ingres or Pentaho). Kalido made the list, with my endorsement, their silly I-CASE like marketing messaging notwithstanding.

Speaking of imperfections — there only are a few category names, and so category assignments can be pretty bizarre. (In an ideal world, middleware wouldn’t be included under “enterprise applications”.) Greenplum hasn’t really “extended” its DBMS with a “cloud” option. As much as I’d like Netezza to be more influential than SAP, that’s probably not the best way to rank them. And there are a number of “This company is on a roll!” kinds of comments that I wouldn’t necessarily endorse.

But those are all nitpicks. On the whole, it’s another nice job.

The Bottom Line  – SAP’s Turning The Corner

Credit must be given to SAP for charting a new course.  A shift in the management philosophy and product direction will take years to realize, however, its not too late for change.  SAP must remember its roots and become more German and less American.  The renewed focus must put customer requests and priorities ahead of SAP’s bureaucracy.  The emphasis must focus on the relationship.  When that reemerges in how SAP works with customers, partners, influencers, and its own employees, SAP will be back in good graces. In the meantime, its  time to get to work and deliver.  Oracle’s Fusions Apps are coming soon and competitors such as IBM, Microsoft, Epicor, IFS, and SalesForce.com will not relent.

I recall the 1980s, when SAP’s main differentiator, at least in the English-speaking US, was a total commitment to customer success, and when it could be taken for granted that SAP would do business ethically. Things change, and not always for the better.

Anyhow, the reason I’m highlighting Ray’s post is that he makes reference to a number of interesting SAP-cetric technology trends or initiatives. In no particular order, Ray suggests:

• SAP’s and Oracle’s (Fusion) efforts to meld memory-centric analytics with operational apps will be crucial for large enterprises — but perhaps only around the middle of the next decade. (I basically agree, although I’d note that:
  • Wisely, Ray suggested a very long time frame.
  • BI/operational app integration has been, on the whole, glacial.
  • The idea that you have to put pre-built aggregates into RAM to get performance is an indictment of market-leading RDBMS — but it’s a fair indictment.
  • I’m not sure whether memory-centric OLAP will wind up in RAM or Flash. If the data stores are updated at near-transactional speeds, RAM may make more sense. Otherwise, Flash should have major advantages.)
• SAP’s long-standing attempts to support third-party development of SAP add-ons are a technological mess, in line with my fears a couple of years ago. However, the business-relationship part of the effort is vastly stronger.
• As SAP focused more on the mid-market, it is partnering closely with Microsoft. (If you think about it, that makes all kinds of sense.)
• Energy/environmental/safety tracking — i.e., sustainability — tools are a big deal. (See also The Economist on that point.)

For example, Greg Rahn twittered to me that Oracle Exadata commonly outperforms existing Oracle installations by a factor of 50 or better, based on a “geometric mean”. What I presume he meant by that is:

• At any one user installation, a number of queries were compared on new system vs. old.
• In each case, the ratio between new and old running time was taken.
• The geometric mean of all those ratios was computed.

Math note: Reversing the order of the second and third steps doesn’t change the outcome at all. Either way, you wind up multiplying N things together, dividing by the product of another N things, and taking the Nth root of all that.

Looking just at the arithmetic, a straightforward geometric-mean approach is not a terrible methodology. Theoretically, I’d prefer to just add up the running times for the whole workload — using each system — and divide the two aggregates. But I’ve tested that change in a couple of cases, and it didn’t seem to make a big difference. In particular, the geometric is better than the arithmetic mean, which gives huge weight to the most extreme number(s) in the set. (SAP used to do just that in marketing BI Accelerator, getting huge results because one customer once got a better than 600X speedup on one particular query out of eight or so.)

That said, there are a number of factors that can render such comparisons seriously misleading. For starters, most of these comparisons omit to consider how often each query will be run. (One advantage to my preferred approach — add up total running time before doing any other arithmetic — is that you can easily weight queries by frequency.) Beyond that, especially when a new challenger system is compared to an old incumbent:

• Comparisons aren’t always run on identical or comparable hardware.
• One system might be running an obsolete release of the software.
• Queries may not be equally well written and tuned on both systems.*
• Both systems may not be equally well administered and/or configured.

In addition:

• A small set of queries may not be representative of a whole workload.
• Since complete workloads are rarely tested, concurrency and contention may not be properly modeled.

And last but not least:

• Marketing departments tend to take their best results and claim those are “typical.”

*I suspect that some of the most dramatic speed-ups we see are for queries that are just plain badly written. On the other hand — if you’ve been running your data warehouse software for a few years and still haven’t figured out how to write your queries for decent performance, maybe it’s somewhat too hard to use …

Related links

• Tips for benchmarking an analytic DBMS
• Tips for buying an analytic DBMS

1) good buy for IBM? why or why not?

Yes. The integration of predictive analytics with other analytic or operational technologies is still ahead of us, so there was a lot of value to be gained from SPSS beyond what it had standalone.  (That said, I haven’t actually looked at the numbers, so I have no comment on the price.)

By the way, SPSS coined the phrase “predictive analytics”, with the rest of the industry then coming around to use it. As with all successful marketing phrases, it’s somewhat misleading, in that it’s not wholly focused on prediction.

2) how does it position IBM vs. competitors?

IBM’s ownership immediately makes SPSS a stronger competitor to SAS. Any advantage to the rest of IBM depends on the integration roadmap and execution.

3) How does this particularly affect SAP and SAS and Oracle, IBM’s closest competitors by revenue according to IDC’s figures?

If one of Oracle or SAP had bought SPSS, it would have given them a competitive advantage against the other, in the integration of predictive analytics with packaged operational apps. That’s a missed opportunity for each.

One notable point is that SPSS is more SQL-oriented than SAS. Thus, SPSS has gotten performance benefits from Oracle’s in-database data mining technology that SAS apparently hasn’t.

IBM’s done a good job of keeping its acquired products working well with Oracle and other competitive DBMS in the past, and SPSS will surely be no exception.

Obviously, if IBM does a good job of Cognos/SPSS integration, that’s bad for competitors, starting with Oracle and SAP/Business Objects. So far business intelligence/predictive analytics integration has been pretty minor, because nobody’s figured out how to do it right, but some day that will change. Hmm — I feel another “Future of … ” post coming on.

4) Do you predict further M&A?

Always.

Related links

• Official word from SPSS and IBM
• Blog posts from Larry Dignan and James Taylor
• James Kobelius’s post, which includes the obvious point that Oracle — unlike SAP — has pretty decent data mining of its own
• Eric Lai’s actual article

*Thanks to Dave Kellogg for tipping me off to Plattner’s paper. I only went to two SIGMOD sessions, neither of which was Plattner’s. Nobody actually mentioned Plattner’s talk to me when I was down at SIGMOD.

Perhaps the most interesting part is Plattner’s claim that what’s demanding about OLTP isn’t database updating per se, but rather maintaining aggregates for quick-response analytics. In his main example of that point, Plattner proposes a real-life “more than 18″ table schema, of which 2 are base tables, and (most of?) the rest are materialized views that his proposed database architecture dispenses with (because analytic performance is sufficiently good without them). Thus, Plattner’s core columnar argument seemingly is

columnar –> natively fast analytics –> no need to maintain aggregates –> much lower update burden.

That said — if Plattner’s paper contained a clear statement of how much more expensive it is to insert or update a single row in a columnar vs. row-based system, I overlooked it. Instead, Plattner seems to be arguing that the volume of base-table updates is low enough that — whatever it may be — column-store update overhead is an acceptable price to pay.  (At one point he claims that only 5% of the data inserted in a financial application ever gets changed.) That may actually be true in a financial accounting system, but seems more questionable in a sufficiently large application that gets its updates from automatic devices, or from the consumer web.

Other highlights include:

• Like most modern observers, Plattner believes Postgres-style timestamping beats update-in-place.
• Plattner also offers a less common reason for liking timestamped inserts over updates-in-place — he thinks timestamps are helpful in planning-oriented applications. In particular, he wants timestamp-aware SQL extensions.
• Plattner claims columnar designs have a 10:1 compression advantage over row stores — specifically 20X vs. 2X — at least using compression schemes that allow for updating at reasonable speed. That seems exaggerated.
• Plattner seemed to drop various references to memory-centric structures SAP already uses. (SAP has long done a lot in-memory, in both the OLTP and planning areas. Years ago SAP told me of a customer that was buying >1 TB of RAM just to run SAP’s planning software. SAP also bragged that >99% of transactions never hit disk, in some sense of “transaction”. )
• There are lots of references to “tenants”, SaaS, and/or SAP’s SaaS product line. So SaaS is evidently a design point. That makes sense. First, SaaS is one of SAP’s biggest vulnerabilities. Second, the toughest customization a SaaS customer might want is to add a few columns to standard tables, which might be easier to accomodate with a columnar approach.

But Sara Lee has not yet decided to purchase the software, and there are substantial underlying tasks to perform as well, he added.

“This is giving us the horsepower [to analyze data] but we need to have harmonized and structured data underneath it.”

This is from the leading test user of the product?

Business intelligence and the associated data management processes need to be reimagined, and I’m increasingly coming to suspect that the big BI conglomerates aren’t up to the task.

Natural language and classic science fiction

Actually, there’s a pretty well-known example of BI near-perfection — the Star Trek computers, usually voiced by the late Majel Barrett Roddenberry. They didn’t have a big role in the recent movie, which was so fast-paced nobody had time to analyze very much, but were a big part of the Star Trek universe overall. Star Trek’s computers integrated analytics, operations, and authentication, all with a great natural language/voice interface and visual displays. That example is at the heart of a 1998 article on natural language recognition I just re-posted.

As for reality: For decades, dating back at least to Artificial Intelligence Corporation’s Intellect, there have been offerings that provided “natural language” command, control, and query against otherwise fairly ordinary analytic tools. Such efforts have generally fizzled, for reasons outlined at the link above. Wolfram Alpha is the latest try; fortunately for its prospects, natural language is really only a small part of the Wolfram Alpha story.

A second theme has more recently emerged — using text indexing to get at data more flexibly than a relational schema would normally allow, either by searching on data values themselves (stressed by Attivio) or more by searching on the definitions of pre-built reports (the Google OneBox story). SAP’s Explorer is the latest such view, but I find Doug Henschen’s skepticism about SAP Explorer more persuasive than Cindi Howson’s cautiously favorable view. Partly that’s because I know SAP (and Business Objects); partly it’s because of difficulties such as those I already noted.

Flexibility and data exploration

It’s a truism that each generation of dashboard-like technology fails because it’s too inflexible. Users are shown the information that will provide them with the most insight. They appreciate it at first. But eventually it’s old hat, and when they want to do something new, the baked-in data model doesn’t support it.

The latest attempts to overcome this problem lie in two overlapping trends — cool data exploration/visualization tools, and in-memory analytics. Tableau and Spotfire are known more for the former; hot BI vendor QlikTech is known for both. And many vendors — established or otherwise — are going to in-memory OLAP.

Collaboration and communication

The reason I’m finally buckling down and posting on this subject is the announcement of Google Wave, which I think foreshadows a revolution in communication and collaboration technology. Google Wave augurs two primary advances. First, it shows how to make email, instant messaging, microblogging, and so on much more useful. Second, Google Wave could evolve in a way that — finally — makes it truly practical for end-users to set up ad-hoc mini-portals that combine arbitrary URL-possessing resources, exposed to arbitrary workgroups of people.

If and when both of those promises are fulfilled, it will become vastly easier for people to reason together about analytic questions. That may take a little while, as Google Wave obviously wasn’t designed with business intelligence in mind. But whether from Google or from a frightened Microsoft redoubling its SharePoint efforts, there’s hope that we’ll see a leap forward in general collaboration technology. And since BI vendors are doing a generally decent job of exposing queries, charts and so on as portlets, it seems likely that business intelligence will benefit from the collaboration arms race.

That’s important. The first time I heard that reporting was as important for communication as for analytics was from Pilot Software a quarter-century or so ago, and it’s just as true now as it was then. In its first incarnations it probably will be a little too dumb for my tastes, focusing more on mindless reporting and same-old KPIs than on deeper analysis. Still, it’s a move in a good direction.

Other directions

As I said at the beginning, I find it too daunting to try to cover all facets of this subject in one post. So I’ll leave out, at a minimum:

• Data warehousing performance and TCO, which I of course write about extensively
• Complex event/stream processing, which I’ve written quite a bit about too
• Data mining and predictive analytics
• Operational BI

plus some hobby horses you probably don’t want to hear about anyway until I work out a better way of articulating my opinions.

But by all means please comment on what I’ve left out just as vigorously as on what I’ve included. This post is just the first of many to come.

]]> http://www.dbms2.com/2009/05/30/reinventing-business-intelligence/feed/ 13 http://www.dbms2.com/2009/04/28/the-sapteradata-deal-explained/ http://www.dbms2.com/2009/04/28/the-sapteradata-deal-explained/#comments Tue, 28 Apr 2009 04:40:32 +0000 Curt Monash http://www.dbms2.com/?p=767 When I first saw the press release about the latest SAP/Teradata deal, I thought it sounded very Barney. But it turns out there’s a little bit of substance, as well. Amazingly, SAP BW doesn’t really run on Teradata right now. This deal will fix that. The time frame seems to be that SAP-BW-on-Teradata will ship with SAP BW 7.2 whenever that goes out. (First half of 2010?) Early adopters may be able to get their hands on it as early as Q3 2009.

Note: It surely would be more precise to insert “NetWeaver” a few times into that paragraph.

Just to be clear — I still don’t see this as a big deal. It doesn’t portend any grand SAP/Teradata joint mission to smite Oracle, IBM, and/or Microsoft. Nor is it a telling first step toward an SAP/Teradata merger. It just removes a particular competitive disadvantage Teradata had vs. Oracle et al., from which Teradata’s smaller specialist competitors still suffer. And it offers SAP BW customers another high-quality DBMS option.

]]> http://www.dbms2.com/2009/04/28/the-sapteradata-deal-explained/feed/ 0 http://www.dbms2.com/2009/03/31/somebody-is-spreading-teradata-acquisition-rumors-again/ http://www.dbms2.com/2009/03/31/somebody-is-spreading-teradata-acquisition-rumors-again/#comments Tue, 31 Mar 2009 17:42:58 +0000 Curt Monash http://www.dbms2.com/?p=735 An mass email from Tom Coffing was forwarded to me today that starts:

I have heard from reliable sources that both HP and SAP have purchased more than 5% of Teradata stock.  My sources tell me that both companies appear to be positioning themselves for a bid.

I got my version of the same email from Coffing yesterday with a different introduction but otherwise the same substance (he’s pushing a new product of his). It also had a different From address.

Possible explanations include but are not limited to:

• Coffing knows something (seems unlikely, but I haven’t actually checked www.sec.gov to confirm or disconfirm)
• Coffing thinks he knows something
• Coffing just made this up (I hope not)
• There’s an April Fool’s Day prank going on (not by me — after my bizarre March, I’m recusing myself from April Fool’s pranks this year)

• Oracle has offered compression since 9i Release 2. It’s token/dictionary on columns or groups of columns. Oracle says 2-3X compression is common and 8X not unheard of. Also, Oracle has always had variable-length rows, which can make its data somewhat more compact to start with than some other vendors’.
• Interestingly, Oracle executes SQL statements directly on compressed data.
• In general, Oracle says its compression has very little overhead.
• Oracle has offered transparent encryption since 10g Release 2, on whole columns or individual tables. I didn’t ask about encryption performance.
• In 11g Release 1, Oracle rearchitected its LOB (Large OBjects) structure for performance, and perhaps functionality as well. Andy Mendelsohn believes performance now is the same as that of raw file systems. Supported features include encryption, compression, and deduplication. Applications for this new functionality include content management such as video or images and — which I found surprising — spatial data. But it’s not so relevant to text and OLAP data, even though technically those are stored in LOBs as well.
• Actually, Oracle stores OLAP in a true MOLAP array. (I recall that that integration took years.)
• Speaking of Oracle’s geospatial functionality, it’s used heavily in ERP. (I didn’t probe for details.) Oracle also says “all” the mapping vendors are Oracle customers (partners?).
• In Ray Roccaforte’s view, the two greatest drivers of data warehouse growth are consolidation (merger-related or otherwise) and web data.
• Oracle believes that, at least among general-purpose DBMS, its main product has the only bitmaps actually stored on disk, and by far the best star optimizations.
• While conceding that table scans are important, and that Oracle had bottlenecks in that regard Exadata is designed to fix, Ray insists there are some applications for which star optimizations really are needed.
• Query pipelining is limited in Oracle, and the optimizer is not geared to optimize streams of queries.
• However, Oracle’s optimizer is sufficiently self-aware to notice when a query runs long and try to do things differently “next time.” For example, it might do more sampling if the statistics proved unreliable, or might take the time to search a bigger solution space.
• Oracle says it already does a lot of CEP-like things to refresh overlapping materialized views simultaneously. (That makes sense.) Oracle is working on truer complex event processing.
• Complexity data point: Oracle’s and SAP’s application suites each have >100,000 tables and >1,000,000 distinct SQL statements.

There were a couple of notes about text analytics as well, but I’ll blog about those separately.