• “Deliver products that really meet customers’ desires and needs.”
• “Successfully convince them that you’re doing so …”
• “… at an attractive overall cost.”

Vendors mentioned here are Oracle, SAP, HP, and IBM. Anybody smaller got left out due to the length of this post. Among the bigger omissions were:

• salesforce.com (multiple subjects).
• SAS HPA.
• The evolution of Hadoop.

A (lingering) issue for SAP and Oracle alike

As I noted in January of this year, integration of business intelligence into operational apps is making very slow progress. Even so, it’s a huge part of the apparent strategy at SAP and Oracle alike, as well it should be. Much of the benefit from automating routine desk work has already happened. The areas ripest for exploitation are the ones where analytics are part of the equation.

Given the lack of tangible progress, why do I think this is a genuine area of Oracle and SAP emphasis? Three reasons of many are:

• Why else did SAP buy Business Objects?
• If they’re not trying to integrate operational apps and analytics, why else does SAP’s emphasis on HANA make sense?
• Without business intelligence in the picture, how does Oracle’s integrated-stack story promise any direct user benefits?*

*As opposed to IT concerns — integration, administration, TCO (Total Cost of Ownership), etc.

After so many years of disappointment, I’m not going to forecast 2012 as a pivotal year for the integration of business intelligence into operational applications. But if one of SAP or Oracle ever does get a significant BI/operational app integration advantage over the other, it could be a major competitive advantage in those application market segments that are still up for grabs. It also is an opportunity for both vendors to gain BI market share in their respective application customer bases.

A more urgent issue for SAP

SAP has put huge amounts of credibility on the line for HANA, the integration of two different and not particularly mature in-memory database technologies. So far, it is difficult to find evidence that HANA is robust enough for widespread adoption. Whether or not SAP can fix that is a huge open question, which could have significant impact on the course of several technology areas: applications, business intelligence, in-memory DBMS, and maybe even hardware.

Based on current information, which is admittedly partial, I’m a short-term pessimist on HANA. Longer-term, I’m on record as saying that traditional databases will eventually wind up in RAM. SAP will surely get that technology right some day, whether or not the way it does so has anything to do with present-day HANA code.

Four more issues for Oracle

Oracle’s ambitions are near-endless, and so also therefore is its list of execution challenges. Four in the analytics area that I find particularly interesting are:

• True hybrid columnar DBMS. I was guessing that Oracle, like Teradata, would announce true hybrid columnar the week of Oracle OpenWorld. I was wrong. But if Oracle can’t bring out true hybrid columnar DBMS functionality relatively soon, Exadata will lose credibility as a competitor to more specialized analytic DBMS.
• Oracle Exalytics. With Exalytics in the mix, Oracle’s technology stack has HANA-like potential. But will Exalytics even ship in 2012? (I think so.) Will it be good for much in the first release? (I’m skeptical.)
• Oracle’s Big Data Appliance. I’m skeptical both about Oracle’s NoSQL product — a favorable InfoWorld review notwithstanding — and Hadoop appliances. But if I’m wrong, and Oracle can successfully embrace/extend the new non-relational paradigms, then it really might regain control over the evolution of data management.
• Oracle’s Endeca acquisition — will Oracle prove me wrong and integrate Endeca effectively into its overall analytic product line? If it does, we might finally see effective text (and eventually speech) navigation of enterprise software. (But as with all Oracle issues cited here, this is something that probably won’t amount to much in 2012 even if it does later go well.)

Three issues for IBM

Like Oracle, IBM is a huge company with many ambitions and hence many execution challenges. The biggest of those is surely: How effective can IBM be at selling outside its existing customer base? I don’t hear as much competitively about IBM DataStage, IBM SPSS or now IBM Netezza as I did when their vendors were independent companies. Even Cognos may not be much of an exception to the rule, although it has its own large customer base outside of IBM’s traditional one. (To lesser extents , the same is of course true of Netezza and numerous other IBM acquisitions.)

Another general issue for IBM is substantively integrating its various product lines, at least to the extent that makes sense. DB2/Netezza integration sounds good, but even that is a matter more of product marketing (the admirable part of that discipline) more than of actual technology. Other integrations (e.g. Cognos/DB2 in various bundles) have tended toward the dubious side.*

*I’m still waiting for IBM to get back to me with examples of how Cognos/DB2 joint tuning amounts to anything. It’s been more than a year, so I’m glad I didn’t hold my breath.

In a somewhat narrower vein, I wonder: Will IBM be able to gain traction for InfoSphere Streams? And if so, when and where will the traction be?

Will HP screw up Vertica?

Vertica has a very attractive product offering. It’s perhaps the most scalable analytic DBMS outside of Teradata, running on the hardware of your reasonable choice.  It’s also the one I recommend most often to clients in the 1-50 terabyte range.

So far HP doesn’t seem to have done much to leadfoot Vertica. (About all I’ve heard from competitors is that Vertica seems to have faded somewhat in the financial services market, and there could be multiple explanations if that is indeed true.) But if HP Vertica does somehow manage to botch things, opportunities will open up for a range of columnar analytic DBMS competitors.

Ultimately, there are six useful things you can do with analytic technology:

• You can make an immediate decision.
• You can plan in support of future decisions.
• You can research, investigate, and analyze in support of future decisions.
• You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
• You can communicate, to help other people and organizations do these same things.
• You can provide support, in technology or data gathering, for one of the other functions.

Technology vendors often cite similar taxonomies, claiming to have all the categories (as they conceive them) nicely represented, in slickly integrated fashion. They exaggerate. Most of these categories are in rapid flux, and the rest should be. Analytic technology still has a long way to go.

In more detail: 

• You can make an immediate decision.
  • The decision can be made by:
    • A machine, and then also executed by a machine, for example in:
      • Web site personalization.
      • Algorithmic trading.
      • Network security and/or load balancing.
    • A machine, and then executed by a human, for example in a call center.
    • A human looking at a machine.
      • This case can be a lot slower than the others.
      • Analytic-operational app integration can be a special case of this, but progress is slow — today’s reality resembles what I proposed in a 2004 white paper.
  • Technologies supporting immediate decision making include:
    • SQL, most commonly.
    • Extensions that are getting added into SQL DBMS, such as in-database model scoring.
    • Other search and query languages.
    • Complex event processing (CEP), whether rules-based or SQL-like.
    • Other rules engines, rarely.
• You can plan in support of future decisions.
  • Technologies that support planning include:
    • Microsoft Excel, first and foremost.
    • Budget-centric tools.
    • Forecasting.
  • Planning hasn’t advanced as well as one would have hoped.
    • The integration of planning and business intelligence has been uninspiring, a couple years of aggressive marketing last decade — especially by Cognos — notwithstanding.
    • Specialty planning languages always seem to disappoint. River Logic is a small vendor with a great idea — which it hasn’t advanced rapidly since the 1990s. This is sadly typical.
  • One bright spot, however, has been demand forecasting.
• You can research, investigate, and analyze in support of future decisions.
  • I’ve just started calling this area investigative analytics.
  • In doing so, I am conflating several disciplines:
    • Statistics, data mining, machine learning, and/or predictive analytics. (Note: I can’t get excited about the distinctions between those closely overlapping technology categories — apologies to Sam Madden and others who do seem to care.)
    • The more research-oriented aspects of business intelligence tools:
      • Ad-hoc query.
      • Drilldown.
      • Most things done by BI-using “business analysts.”
      • Most things within BI called “data exploration.”
    • Analogous technologies as applied to non-tabular data types such as text or graph.
  • There’s a lot further to go.
    • It’s still very early days for in-database analytic technology.
    • Any two of statistics, business intelligence, and text analytics could be much better integrated with each other than they are.
• You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
  • The guts of business intelligence — reports and dashboards — are really monitoring tools.
  • Monitoring is the jumping-off point for a lot of decision making, planning, and investigation. First you notice the anomaly or need, then you set out to do something about it.
  • I think this technology could use a lot of improvement.
• You can communicate, to help other people and organizations do these same things.
  • Since the dawn of reporting, reports have used as much to communicate among colleagues as they have to truly support personal decision-making.
  • BI vendors have done decent jobs in recent years of advancing the communication aspects of BI, in two respects:
    • General share-ability of reports and the like.
    • Stakeholder-facing BI.
  • But more profound BI-centric collaboration is advancing too slowly.
• You can provide support, in technology or data gathering, for one of the other functions.
  • Well, duh. That’s most of what I write about in this blog, especially in the areas of DBMS and ETL/ELT/ETLT.

• Clearing some underbrush.
• Speculation about what IBM/Netezza will do.
• Speculation about alternative acquirers for Netezza.
• Speculation about what IBM/Netezza competitors will do.

First, the underbrush:

• Apparently there’s a deal breakup fee of around 3% of the purchase price. So yes, it’s perfectly conceivable that IBM get outbid.
• I’ve been getting some DB2 briefings, which is why I’ve blogged about some specialized technical points from same. But I can’t yet say why the theoretically great-sounding data warehousing option DB2 isn’t more successful in the market.
• I haven’t been briefed in any detail on IBM’s more general analytic appliance stack, but I get the impressions:
  • To date it’s more packaging than technology.
  • However, some performance optimizations have already happened (e.g. Cognos/DB2).
• Lots of analytic DBMS vendors told me about partnership discussions or actual partnership activity with IBM, perhaps via IBM’s service arm and/or in specific geographies. So IBM acquiring one of them was never inconceivable.
  • And of course IBM has multiple DBMS products (most notably including the former Informix) in any case.
• The fact that Netezza currently happens to use IBM hardware for most copies of its TwinFin appliance is pretty irrelevant.
  • In fact, the alternative, hardware from NEC, was probably a little better for the job. That’s irrelevant too.
• I hate it when my biggest clients get acquired, but it was pretty inevitable. Consolidation is happening.
  • In particular, Netezza seemed to be somewhat more natural as a seller than a buyer, and didn’t happen to be super-aggressive on buying.
  • By the way, the rumors of an almost-happened Oracle/Vertica deal seem pretty credible. I don’t know exactly when this was, but I’m guessing it was before Chris Lynch started as CEO.
• Netezza had multiple positionings, which would have been unsustainable or at least under-optimal over time without some kind of M&A activity, including:
  • Simple, easy, low-TCO appliance. (This has been Netezza’s core positioning from the getgo.)
  • Super-capable analytic engine.
  • Enterprise data warehouse (EDW) for medium-sized enterprises. (E.g. — I think — Ross Stores.)
• Netezza was really focusing on $1/2 million and up sales, except for some OEM appliance deals.
  • Netezza’s OEM appliance efforts never struck me as making a whole lot of sense, and the list of OEM partners didn’t amount to much, possibly excepting ones whose real motivation was to sell into the Netezza market.
• Netezza’s detailed attacks on Exadata highlight differences between Netezza and one-size-fits-all approaches, and hence serve to remind us that Netezza/DB2 integration wouldn’t be particularly natural.

Now, the speculation about the technical and/or business evolution of Netezza and IBM data warehousing, assuming IBM indeed winds up being the Netezza buyer:

• Netezza’s products are too successful and far along for IBM to shut down. This is not at all a Kickfire/Teradata situation.
• I hope and trust IBM will get Netezza onto a more frequent and consistent hardware revision cycle, starting in, say, the second half of 2011.
• Oopsie on the sometimes close, sometimes not-so-close Netezza/SAS relationship. It looks like Netezza/SPSS is the main focus going forward.
• More generally, Netezza’s high-end analytic positioning should be strengthened by IBM ownership. Netezza’s approach to advanced analytics is more “kitchen sink” than tightly-designed integration, so mixing together Netezza and IBM technology should improve both.
• However, the opposite kind of design considerations apply to Netezza’s core DBMS functions. When it comes to database processing, Netezza’s design is carefully integrated. Surely IBM can send over some engineers to teach Netezza some things it has figured out, but for most purposes DB2 and Netezza will remain as separate technologies, not integrated with each other.
• However, I expect IBM to totally revamp Netezza’s data movement and integration. IBM is a huge player in data movement and integration. Netezza has been struggling even with its modest goals in the area.
  • And by the way, if DB2 and Netezza are to remain as separate product lines, it would surely be to IBM’s advantage to make it easy to allocate data among them as makes sense.
  • As just one piece of that, IBM needs its own version of EMC/Greenplum Chorus.
• I don’t yet have an opinion as to whether IBM’s primary full-stack analytic appliance strategy will be based on DB2, Netezza, or both.
• However, pushing Netezza technology harder for OEM deals would make good sense.
• Both IBM and Netezza believe in solid-state memory for data warehousing. Expect the merged company to be aggressive in that area.

So who else could bid for Netezza?

• HP would be an obvious Netezza bidder, in that HP clearly needs to change its DBMS appliance strategy. However, HP is a little busy changing CEOs.
• Any big acquirer would add value to Netezza via the message “OK, the shoe has dropped, they’re not going away, they have staying power, etc.” But that’s all Dell would add, and I question whether that’s worth a bidding war. Besides, Dell wants to partner with as many vendors as possible.
• NEC wouldn’t add much more. Netezza is already doing great in East Asia anyway, from what they tell me, and Japanese enterprise computer companies have poor track records outside their home region.

So where does the analytic DBMS market go from here?

• Further consolidation is inevitable. The analytic DBMS market will not indefinitely support all of Oracle, IBM, Microsoft, SAP/Sybase, Teradata, Vertica, Aster Data, Infobright, and ParAccel. Yet each of those (I think even ParAccel) has a legitimate reason to continue business and product operations. And that’s even before mentioning Kognitio, SAND, Kx, and so on, plus of course HP.
• Over the next years, different styles of analytic DBMS engine could usefully merge.
  • Row-based/columnar integration for disk-based systems is a pretty well-established trend now.
  • SAP’s fuss about in-memory column store is getting me-too response from Oracle. Use your imagination about how that idea could be extended, and you see that all that stuff is still in its early days.
  • And that’s before we even mention XML documents, graph, etc.
• Of all that, the part that would most naturally drive analytic DBMS vendor consolidation (as opposed to, say, tuck-in acquisitions from adjacent sectors) would be row-based/columnar integration. For example:
  • Oracle is already known to have looked at buying Vertica. ParAccel could be cheaper. Infobright was once a highly strategic partner for MySQL.
  • If HP can’t bring itself to admit it’s killing Neoview, a columnar acquisition could be a face-saver.

1) good buy for IBM? why or why not?

Yes. The integration of predictive analytics with other analytic or operational technologies is still ahead of us, so there was a lot of value to be gained from SPSS beyond what it had standalone.  (That said, I haven’t actually looked at the numbers, so I have no comment on the price.)

By the way, SPSS coined the phrase “predictive analytics”, with the rest of the industry then coming around to use it. As with all successful marketing phrases, it’s somewhat misleading, in that it’s not wholly focused on prediction.

2) how does it position IBM vs. competitors?

IBM’s ownership immediately makes SPSS a stronger competitor to SAS. Any advantage to the rest of IBM depends on the integration roadmap and execution.

3) How does this particularly affect SAP and SAS and Oracle, IBM’s closest competitors by revenue according to IDC’s figures?

If one of Oracle or SAP had bought SPSS, it would have given them a competitive advantage against the other, in the integration of predictive analytics with packaged operational apps. That’s a missed opportunity for each.

One notable point is that SPSS is more SQL-oriented than SAS. Thus, SPSS has gotten performance benefits from Oracle’s in-database data mining technology that SAS apparently hasn’t.

IBM’s done a good job of keeping its acquired products working well with Oracle and other competitive DBMS in the past, and SPSS will surely be no exception.

Obviously, if IBM does a good job of Cognos/SPSS integration, that’s bad for competitors, starting with Oracle and SAP/Business Objects. So far business intelligence/predictive analytics integration has been pretty minor, because nobody’s figured out how to do it right, but some day that will change. Hmm — I feel another “Future of … ” post coming on.

4) Do you predict further M&A?

Always.

Related links

• Official word from SPSS and IBM
• Blog posts from Larry Dignan and James Taylor
• James Kobelius‘s post, which includes the obvious point that Oracle — unlike SAP — has pretty decent data mining of its own
• Eric Lai‘s actual article

*By the way, they seem to want use Applix technology, now owned by IBM/Cognos, to do it, more for the planning tools than for the cool in-memory OLAP engine itself. Your mileage may vary.

If you don’t go for fancy slice/dice tools, then do something else to make sure your drilldown or exploration are up to snuff. Just about every enterprise is going to be seeing some distressing numbers right now. But that doesn’t mean every part of every line of business is in equal trouble. Teasing that apart is important at times like this, or employment and investment could get strangled do to overreaction.

I have a small enough business that I can keep things like this in my head, without bothering to run precise three-significant-figures calculations. But I’ll tell you this — my revenue in analytics is sure healthier than my business in OLTP or custom publishing.

And don’t necessarily stop there. We’ve heard the horror stories of investment models failing in this unusual economy. Well, what about your own predictive analytic models? If you have formal models of buyer behavior, what makes you think the future will be like the past? Times have changed. No matter what your usual schedule is, you should be revalidating and perhaps strengthening your more important models now. If that overtaxes your infrastructure, and you can’t afford the capital investment to do something about that — well, appliances are pretty cheap, and some SaaS data warehousing offerings are (at least in the short term) even cheaper.

February, 2011 edit: Here’s a partial link that works right now.

Of course, some parts do make sense.  E.g.:

• Business Objects’ completeness of vision seems to have been downgraded because of its new affiliation with SAP’s ever-confused Netweaver strategy.
• Microsoft’s completeness of vision is dinged for — well, for not being very complete.
• SAS, which unlike other vendors actually gets customers to integrate BI and predictive analytics, gets top marks in “completeness of vision”.
• IBM/Cognos leads the way overall.

Parts I find more dubious include:

• Whether or not vendors have strong international sales presences affects their “completeness of vision” scores. Huh?
• In-memory analytics are hugely emphasized, to the point that TIBCO Spotfire gets very high “completeness of vision” scores despite being just a portion of an overall BI product line. Yet vendors who get similar performance from allowing drilldown within reports don’t seem to get the same credit.
• Endeca isn’t included, while Spotfire is.
• Despite criticizing Microsoft for not delivering on promised products and Oracle for not doing much at all, Gartner gives both better “ability to execute” marks than are given to Information Builders and Microstrategy.
• While Gartner correctly points out in the commentary that company size is not a strong indicator of ability to execute, this awareness doesn’t seem to have been reflected in the actual chart.
• Gartner’s supposedly rigorous numbers seem sloppy. LogiXML is seemingly cited as almost making the $20 million product revenue cutoff, despite being a company with $7.3 million in overall revenue.

I don’t doubt that Gartner has done good research in support of this article. Indeed, I learned things from reading the supporting commentary. But the actual Magic Quadrant presentation methodology is, as always, fatally flawed.

Also working on Cognos’s behalf during this period was lobbyist Richard McDonough, another close friend of DiMasi’s, who was paid hundreds of thousands of dollars to help the company secure state work. He failed to report more than $300,000 in lobbying fees until a Globe story earlier this month detailed his extent of his relationship with Cognos.

Related links

• Cognos/State of Massachusetts scandal
• Another Cognos scandal in Massachusetts

The Globe now reports that a multimillion dollar deal the prior year with the Massachusetts Department of Education was equally dubious. Lowlights include:

A former state Department of Education administrator has told investigators that she was offered a private sector job by a sales representative for Cognos ULC, around the same time that the software company was attempting to win a lucrative education contract in 2006, according to officials briefed on the matter.

While pushing to land the smaller education department contract, Lally specifically emphasized that he could have money added to the state budget to fund the deal because of his friendship with DiMasi, according to Chew and a former Cognos employee. The contract provided for a data warehouse system, which would allow the department to collect, track, and share data about students, teachers, and finances across the state.

In April 2006, Lally’s prediction proved true. House lawmakers added a new $5.2 million line item for the contract through a House budget amendment. DiMasi told fellow legislators at the time that the amendment was a priority, according to an official with direct knowledge of the budget negotiations.

“The two finalists – we spent a long period of time putting them through their paces,” Bickerton said.

That period of time: two days, according to a timeline provided by the Department of Education.

The story is this: Cognos sold 20,000 seats of software to Massachusetts for $13 million. There were technical violations of purchase procedures, and other aspects of the deal that didn’t pass the smell test. After IBM bought Cognos, the deal was rescinded, and is being rebid.

As per the Boston Globe, highlights of the mess include:

• The Cognos salesman on the deal was well connected politically, specifically with the House Speaker, who seems to have been a prime mover behind there being a BI purchase.
• A much higher fraction of the workforce got seats than is generally regarded as necessary in state governments (25% overall). Ohio, by way of comparison, bought BI seats for 10% of the workforce.Ohio, by way of comparison, bought BI seats for 10% of the workforce.
• The overall price of the deal is much higher than the two alternative bids (SAS and Oracle — no Business Objects or Information Builders, for some reason).
• Prior studies as to what was or wasn’t needed in the BI purchase were ignored. For example, they recommended seats only for 1-10% of its workforce.

IDG has some good coverage too, the highlight being:

For one, the state’s Information Technology Division did not widely advertise the fact it was looking for performance management software, according to the report.

Instead, “a staff member at ITD simply consulted a chart of leaders in performance management developed by the analytical firm Gartner Group and e-mailed the Request for Quotes to four companies identified as ‘leaders.’ “

Three vendors — Cognos, Oracle and SAS — responded to the e-mail, according to the report. ITD staffers developed a scoring sheet containing 104 criteria. The ITD team in charge never finished evaluating the vendors with the sheet, but at the time they stopped Cognos had the high score, with 69.39 points, followed by SAS with 57.38 and Oracle with 27.49, the report states.

The IG’s investigation found that due to a typographical error in the spreadsheet’s formula, the scores for all three vendors were flawed, with many points going uncounted.

Please supply your own punchline.

What has not been reported, insofar as I recall or can find via quick searches on Google or Boston.com, is whether the Cognos software included a lot more capabilities than the other offerings. The obvious candidate would be budgeting/planning. That’s something that governments like and Cognos tries to bundle into large enterprises deals, but which is not a strength of (for example) rival bidder SAS.

• solidDB is actually a very interesting hybrid disk/in-memory memory-centric database management system. However, the press release announcing the deal makes it sound as if solidDB is in-memory only.
• That strongly suggests that IBM is buying Solid mainly to compete with Oracle TimesTen. As of last June, solidDB was already IBM’s TimesTen answer via a partnership; this deal just solidifies that arrangement.
• This probably isn’t good news for Solid’s MySQL engine. That’s a pity, since solidDB technically has the potential to be the best MySQL engine around.
• Notwithstanding IBM’s presumed intentions, Solid’s main market success historically is as an embedded system in telecommunications equipment, network software, and similar systems.
• Last year I wrote a white paper on memory-centric data management, showcasing four products. IBM now has bought two of them, namely Solid’s and Applix’s (via Cognos).
• Comparisons to IBM’s embedded Java DBMS Cloudscape are pointless. That’s just a failed product vs. solidDB or Sybase SQL Anywhere, and IBM long ago cut its losses.

Here’s a link to a webcast I did with Solid back in June.

Edit: As per the invitations to the 11:00 am conference call, the following — straight from the press release — is evidently the official IBM one-paragraph description of the acquisition.

The acquisition of Solid supports IBM’s global Information on Demand strategy by adding real-time data access capabilities to the company’s portfolio of database and information management offerings. Solid’s software uses in-memory database technology to quickly retrieve data from a computer’s memory (or RAM). Using this software, businesses can access and store data at speeds up to ten times faster than traditional disk database systems. Solid’s architecture also enables applications to recover from system failure rapidly and to require little disk space and almost no hands-on administration.

Edit: IDG has quotes confirming the emphasis on TimesTen competition. IDG also reminds us that MySQL 6.0 will have its own storage engine Falcon. I guess that’s pretty much do or die for MySQL now.