• “Deliver products that really meet customers’ desires and needs.”
• “Successfully convince them that you’re doing so …”
• “… at an attractive overall cost.”

Vendors mentioned here are Oracle, SAP, HP, and IBM. Anybody smaller got left out due to the length of this post. Among the bigger omissions were:

• salesforce.com (multiple subjects).
• SAS HPA.
• The evolution of Hadoop.

A (lingering) issue for SAP and Oracle alike

As I noted in January of this year, integration of business intelligence into operational apps is making very slow progress. Even so, it’s a huge part of the apparent strategy at SAP and Oracle alike, as well it should be. Much of the benefit from automating routine desk work has already happened. The areas ripest for exploitation are the ones where analytics are part of the equation.

Given the lack of tangible progress, why do I think this is a genuine area of Oracle and SAP emphasis? Three reasons of many are:

• Why else did SAP buy Business Objects?
• If they’re not trying to integrate operational apps and analytics, why else does SAP’s emphasis on HANA make sense?
• Without business intelligence in the picture, how does Oracle’s integrated-stack story promise any direct user benefits?*

*As opposed to IT concerns — integration, administration, TCO (Total Cost of Ownership), etc.

After so many years of disappointment, I’m not going to forecast 2012 as a pivotal year for the integration of business intelligence into operational applications. But if one of SAP or Oracle ever does get a significant BI/operational app integration advantage over the other, it could be a major competitive advantage in those application market segments that are still up for grabs. It also is an opportunity for both vendors to gain BI market share in their respective application customer bases.

A more urgent issue for SAP

SAP has put huge amounts of credibility on the line for HANA, the integration of two different and not particularly mature in-memory database technologies. So far, it is difficult to find evidence that HANA is robust enough for widespread adoption. Whether or not SAP can fix that is a huge open question, which could have significant impact on the course of several technology areas: applications, business intelligence, in-memory DBMS, and maybe even hardware.

Based on current information, which is admittedly partial, I’m a short-term pessimist on HANA. Longer-term, I’m on record as saying that traditional databases will eventually wind up in RAM. SAP will surely get that technology right some day, whether or not the way it does so has anything to do with present-day HANA code.

Four more issues for Oracle

Oracle’s ambitions are near-endless, and so also therefore is its list of execution challenges. Four in the analytics area that I find particularly interesting are:

• True hybrid columnar DBMS. I was guessing that Oracle, like Teradata, would announce true hybrid columnar the week of Oracle OpenWorld. I was wrong. But if Oracle can’t bring out true hybrid columnar DBMS functionality relatively soon, Exadata will lose credibility as a competitor to more specialized analytic DBMS.
• Oracle Exalytics. With Exalytics in the mix, Oracle’s technology stack has HANA-like potential. But will Exalytics even ship in 2012? (I think so.) Will it be good for much in the first release? (I’m skeptical.)
• Oracle’s Big Data Appliance. I’m skeptical both about Oracle’s NoSQL product — a favorable InfoWorld review notwithstanding — and Hadoop appliances. But if I’m wrong, and Oracle can successfully embrace/extend the new non-relational paradigms, then it really might regain control over the evolution of data management.
• Oracle’s Endeca acquisition — will Oracle prove me wrong and integrate Endeca effectively into its overall analytic product line? If it does, we might finally see effective text (and eventually speech) navigation of enterprise software. (But as with all Oracle issues cited here, this is something that probably won’t amount to much in 2012 even if it does later go well.)

Three issues for IBM

Like Oracle, IBM is a huge company with many ambitions and hence many execution challenges. The biggest of those is surely: How effective can IBM be at selling outside its existing customer base? I don’t hear as much competitively about IBM DataStage, IBM SPSS or now IBM Netezza as I did when their vendors were independent companies. Even Cognos may not be much of an exception to the rule, although it has its own large customer base outside of IBM’s traditional one. (To lesser extents , the same is of course true of Netezza and numerous other IBM acquisitions.)

Another general issue for IBM is substantively integrating its various product lines, at least to the extent that makes sense. DB2/Netezza integration sounds good, but even that is a matter more of product marketing (the admirable part of that discipline) more than of actual technology. Other integrations (e.g. Cognos/DB2 in various bundles) have tended toward the dubious side.*

*I’m still waiting for IBM to get back to me with examples of how Cognos/DB2 joint tuning amounts to anything. It’s been more than a year, so I’m glad I didn’t hold my breath.

In a somewhat narrower vein, I wonder: Will IBM be able to gain traction for InfoSphere Streams? And if so, when and where will the traction be?

Will HP screw up Vertica?

Vertica has a very attractive product offering. It’s perhaps the most scalable analytic DBMS outside of Teradata, running on the hardware of your reasonable choice.  It’s also the one I recommend most often to clients in the 1-50 terabyte range.

So far HP doesn’t seem to have done much to leadfoot Vertica. (About all I’ve heard from competitors is that Vertica seems to have faded somewhat in the financial services market, and there could be multiple explanations if that is indeed true.) But if HP Vertica does somehow manage to botch things, opportunities will open up for a range of columnar analytic DBMS competitors.

This post is a moderately edited form of an actual interview. Two other posts cover analytic trends to watch (planned) and analytic vendor execution challenges to watch (already up).

Question: What do you think will happen next year with the Tableaus of the world?

Answer:

• I think adoption of flexible-visualization business intelligence tools will continue to be rapid.
• I think enterprise-friendly features will be increasingly important as a basis of competition.

Question: What do you mean by “enterprise-friendly”?

Answer: An example would be QlikTech no longer forcing you to use their native ETL, but rather working with Informatica and soon other third-party products. Also important can be:

• Database size.
• Concurrency.
• A full-featured development cycle for analytic applications.

Question: What does HP have to do to be relevant in analytics/data warehousing?

Answer: Avoid stupidity. HP Vertica is already relevant.

Question: OK. But what can HP do to build on Vertica?

Answer: HP — which botched Exadata 1 hardware — could do a good job with SAP HANA or other kinds of appliance products.

However:

• I don’t think trying to force Vertica beyond its natural growth — the way EMC is with Greenplum — is necessarily a good idea. Natural growth in Vertica’s case is plenty fast anyway.
• Obviously, making good Vertica hardware would be nice. But being hardware-independent is crucial to Vertica, not least because of cloud deployment, an option many buyers want to at least have in their hip pockets.

Question: You expressed some skepticism toward mobile BI/use cases. Why so?

Answer: The form factor hurts functionality a lot, so it’s only worthwhile in cases where timeliness is key.

And without more refined alert-setting functionality, it’s hard to think of that many cases.

Note: My views on mobile BI haven’t changed much since July, 2010.

Question: What about the idea of an enterprise being able to pay-per-drink to run jobs on an analytic cluster. Do you expect that concept to have any legs in 2012?

Answer: While other kinds of SaaS (Software as a Service) BI might make sense, remote computing BI that focuses on hardware cost sharing is problematic. Moving data in and out of the cluster is a big part of the overall cost, at least if you plan to process it only occasionally once it gets there. I haven’t seen a plan yet that gets around that point.

• HP needs to make outstanding enterprise systems again.
• They fell away from that target under Mark Hurd, but they surely can hit it again, based on the remnants of DEC (Digital Equipment Corporation), Tandem, the higher-end part of Compaq, and of course the original HP systems group.
• In particular:
  • Rumors say that Oracle Exadata 1 boxes, made by HP, were much lower quality than Exadata 2 boxes made by Sun.
  • HP Neoview was a waste of good engineering talent.
  • I’d like to see a few excellent Vertica appliances.
  • I hope the SAP HANA appliances go well, whenever HANA finally becomes a serious product.
  • The general move from disk to solid-state memory should offer some opportunities.

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays.
• Human/Nontabular data — i.e., all other data generated by humans.
• Machine-Generated data.

I won’t now stand by every nuance in that post, which may differ slightly from those in my more recent posts about machine-generated data and poly-structured databases. But one general idea is hard to dispute:

Traditional database data — records of human transactional activity, referred to as “Human/Tabular data above” — will not grow as fast as Moore’s Law makes computer chips cheaper.

And that point has a straightforward corollary, namely:

It will become ever more affordable to put traditional database data entirely into RAM. 

Actually, there are numerous ways for OLTP, other short-request, and some analytic databases to wind up in RAM.

• SAP has some good ideas for how it could happen, banging transactions into what is essentially an in-memory analytic database. (I dispute SAP’s claims of transformational database technology leadership, but that doesn’t mean the underlying ideas aren’t good.)
• For those who can afford the associated technology disruption, memory-centric object-oriented DBMS could be appealing.
• Web scalability best practices commonly include keeping data in RAM (e.g., that’s pretty much the point of caching layer memcached).
• SaaS (Software as a Service) companies — such as Workday — often bring a particular tenant’s database entirely into RAM.
• QlikView highlights the benefits of doing business intelligence in RAM.
• SAS HPA makes the argument that even “big data analytics” should sometimes be done in RAM.
• I don’t have particularly favorable opinions at this time about marketing strategies or momentum at Oracle TimesTen, IBM solidDB, or VoltDB, but those examples at least serve to illustrate that memory-centric OLTP DBMS have existed for years.
• Actually, SAP has at least two good ideas, if you count Sybase as part of SAP.

And here’s the kicker: Intel told me last year that CPUs are headed to 46-bit address spaces around mid-decade. Indeed, they hired me to help figure out if that was enough.* That multiplies out to 64 terabytes of RAM on a single server, chip costs permitting. So most of what we now think of as operational databases — and many of the analytic ones too — will fit in-memory, even if they run very large businesses.

*And did so without putting the discussion under any kind of NDA.

Likely consequences of all this include:

• Legacy apps will (eventually) be consolidated and virtualized in-memory. Their underlying databases will grow so slowly that eventually the cost of putting them in RAM will be too low to worry about.
• Expensive storage systems will (continue to) be irrelevant to database processing. Databases that don’t fit in RAM will typically be big enough to require the attention of a lot of CPUs — and in those cases the DBMS software itself will handle all the storage tasks.
• Major OLTP DBMS vendors, such as Oracle, will need alternate in-memory code lines, because disk-centric architectures are sub-optimal in-memory. Well, that’s what they have those big R&D budgets for.
• SaaS vendors and web businesses may not rely on today’s major OLTP DBMS vendors. (I was going to say “won’t” rather than “may not” until I recalled the likely M&A endgame.) Traditional enterprises may blanch at migrating away from their legacy DBMS environments, but the trade-offs are different for technology companies using DBMS as subsystems.

Of course, the same trends that make data-storing chips cheaper will make data-generating chips cheaper too. So, just as there are huge amounts of machine-generated data that you’d never pay to store in RAM, the same will still be true 10 years from now; the data volumes involved will just be a lot bigger. And thus there will still be plenty of very large analytic databases using relatively cheap forms of storage, perhaps even disk.

But OLTP and other short-request processing are likely to wind up in-memory. And the same may be true for a considerable amount of analytics, especially but not only if the analytics have a low-latency requirement.

• This is a list of Monash Advantage members.
• All our vendor clients are Monash Advantage members, unless …
• … we work with them primarily in their capacity as technology users. (A large fraction of our user clients happen to be SaaS vendors.)
• We do not usually disclose our user clients.
• We do not usually disclose our venture capital clients, nor those who invest in publicly-traded securities.
• Included in the list below are two expired Monash Advantage members who haven’t said they will renew, as mentioned in my recent post on analyst bias. (You can probably imagine a couple of reasons for that obfuscation.)

With that said, our vendor client disclosures at this time are:

• Aster Data
• Cloudera
• CodeFutures/dbShards
• Couchbase
• EMC/Greenplum
• Endeca
• IBM/Netezza
• Infobright
• Intel
• MarkLogic
• ParAccel
• QlikTech
• salesforce.com/database.com
• SAND Technology
• SAP/Sybase
• Schooner Information Technology
• Skytide
• Splunk
• Teradata
• Vertica

That list includes the two I’m obfuscating, plus one more who just emailed to say a signed renewal contract is arriving this week. It does not include others who, less concretely, have said they will sign up soon.

Also, I guess there’s a bit of a gray area for Tableau. As far as I’m concerned, I’m doing an upcoming co-sponsored webinar just for Monash Advantage member Aster Data. Indeed, I declined to contract with or bill Tableau directly for its share,  because I had no good way to do that paperwork. But even so, Tableau is a cosponsor, was involved in the planning discussions and, behind the scenes, is surely footing part of the bill.

*At the time of this writing, I don’t have a link to a free version of the full report. At the time of this writing, the 2011 Forrester Wave for Enterprise Data Warehouse Platforms graphic can be found here.

One example of the confusion pervading the 2011 Forrester Wave for Enterprise Data Warehouse Platforms lies in a list of three supposed trends.

• The Forrester Wave somehow conflates SaaS and MPP processing, tying them both to the term “cloud.” (In reality, the SaaS/cloud and MPP/cloud equations depend on two rather different word-senses for “cloud”.)
• The Forrester Wave then conflates EDWs, analytic computing systems, and application servers, the latter perhaps because of the “data-application server” product category name Aster Data floated. The Forrester Wave also conflates investigative analytics with low-latency operational processes that exploit investigative analytics’ results.
• The Forrester Wave then conflates social media, “unstructured data” (by which it seems at one point to mean text and at another point to also mean logs), solid-state drives, and a whole bunch of other technologies (especially but not only low-latency ones) into another supposed single trend.

Some of the sillier specific claims in the Forrester Wave for Enterprise Data Warehouse Platforms include:

• According to the 2011 Forrester Wave for Enterprise Data Warehouse Platforms, Netezza has hybrid row/columnar persistence, while most other vendors cited don’t. To recycle an old Larry Ellison joke, somebody obviously has a better pharmacist than I do. It’s tough to imagine how anybody who understands columnar storage could at all believe Netezza currently offers it.
• According to the 2011 Forrester Wave for Enterprise Data Warehouse Platforms, EMC/Greenplum is limited in the hardware it supports. Actually, Greenplum runs on pretty much any commodity Intel hardware, just like any other software-only DBMS does.
• According to the 2011 Forrester Wave for Enterprise Data Warehouse Platforms, Teradata, Sybase, and others are differentiated in their Hadoop support. Actually, Hadoop support of various forms is a checkmark item for analytic DBMS vendors.
• According to the 2011 Forrester Wave for Enterprise Data Warehouse Platforms, Oracle, Teradata, and others are differentiated in their cloud/SaaS support. Actually, having some kind of public cloud offering is a checkmark item; use of same is quite a different matter.
• The 2011 Forrester Wave for Enterprise Data Warehouse Platforms calls out EMC Greenplum for special praise in mixed workload management. Greenplum will probably be fine in concurrency and workload management, but implying it’s a leader is overstated.
• According to the 2011 Forrester Wave for Enterprise Data Warehouse Platforms, Vertica has not made a significant investment in real-time technologies (despite doing a lot of work with StreamBase and selling a lot into the algorithmic trading market). I disagree.
• Also according to the 2011 Forrester Wave for Enterprise Data Warehouse Platforms, Vertica has not made a significant investment in in-memory technology, despite the fact that all its updates pass through Vertica’s in-memory, query-responsive “Write-Optimized Store.” I disagree.

Even leaving aside the errors that obviously riddled the Forrester Wave for Enterprise Data Warehouse Platforms’ underlying 56-row matrix, I dispute the whole premise of the exercise. I’m not a big fan of overarching scorecard-based rankings, because the right choice of product varies so much by use case. For example:

• If you’re a smallish enterprise who can realistically do OLTP and data warehousing on the same instance of your DBMS, Oracle and Microsoft blow away everybody else mentioned.
• If columnar compression methods work really well for your use case, Vertica or maybe Oracle Exadata might shine.
• If you typically only retrieve a few columns from a wide table, so that columnar I/O is what you care most about, Vertica, Sybase, or even EMC Greenplum might shine. (The decidedly non-columnar Netezza and Oracle Exadata approaches to predicate pushdown might or might not excel as well.)
• If your database is above a certain size, some of the alternatives (such as Sybase IQ or non-Exadata Oracle) should be taken off the table.
• If you have a highly concurrent mixed workload, nobody else is as proven as Teradata.
• If you don’t want to invest much in database administration, Oracle is about the last vendor you should consider, and Netezza might be the first.

More excusable is some terminological confusion in the Forrester Wave for Enterprise Data Warehouse Platforms, the essence of which is this:

Notwithstanding its name, the Forrester Wave for Enterprise Data Warehouse Platforms isn’t just talking about what are called enterprise data warehouses (EDWs), but rather a broader range of analytic database management systems and use cases. These include:

• What are classically called operational data stores (the focus on “Next-Best Actions” suggests those are included).
• Analytic platforms/analytic computing systems (the high-level mentions of MapReduce, predictive modeling integration, and so on suggest they’re in too).
• Reporting data marts (some of the vendors cited might not make the minimum count threshold unless those are included too).

Indeed, the definition provided of “EDW” basically boils down to “runs SQL, is tuned in some way for analytics, has a cost-based or other query optimizer, and isn’t tied to a specific application.”

Frankly, I think classical EDWs have their problems, and are not necessarily the best way to address the numerous use cases for analytic DBMS technology. And product category names are commonly problematic anyhow. So I don’t much mind this overloading of the EDW term. But in one respect I think the Forrester Wave overdoes its inclusiveness — it includes things that aren’t actually DBMS, and then marks down just about every product cited for being a real DBMS rather than some sort of above-DBMS layer, at least when those things are sold by SAP. I’ve never agreed with the idea that SAP’s BW/BWA products should be included in a comparison with the other products cited in the Forrester Wave at all, and SAP HANA doesn’t change my mind.

One last thing — I’m suspicious of the Forrester Wave for Enterprise Data Warehouse Platforms’ comments on data warehouse appliance prices. However, they are hard to judge without knowing whether Forrester was using the term “raw data” in its usual sense, or actually means “user data”, and also without knowing whether Forrester is talking about list or “street” pricing.

• Neil Raden points out that business intelligence dashboards can be dangerously misleading. His reasoning (sound) is that whatever you measure is apt to be distorted by the fact people know they’re being measured. His solution (implied) is to hire a good-looking consultant like himself to do it right.
• I’ve had my issues with Fred Holahan, who was VP of Marketing when I posted that EnterpriseDB was not to be trusted. (That said, Fred is long gone from EnterpriseDB and my opinion hasn’t changed.) But he’s put up a good series of posts on the basis of the open source “progressive engagement” marketing funnel, including this gem on why you shouldn’t count on monetizing your community/free users.
• Oracle plans to increase its acquisition budget. The figure given is $70 billion over the next 5 years. Edit: But see this funny Register followup.
• Clayton Christensen wrote a phenomenal article on how to live a good life, from a very business-y perspective. (Only in one anecdote was it too religiously-oriented for my tastes.) Takeaways include:
  • Your core goals probably revolve around something other than business success. (E.g., family.) Don’t lose sight of that.
  • To the extent you’re a manager or leader, you may have a huge impact on other people’s lives. Use that power in admirable ways.
  • Teach people how to fish for answers, rather than just giving them answers. They’ll probably come to better conclusions than you would have anyway. (This is a core principle in my own consulting.)
  • Take time to reflect. And by the way, the same techniques you use for strategic analysis in business can be applied to your life as well.
• Mark Suster has a pretty good post expanding on my first Christensen takeaway, highlighting a point too often missing from articles in that genre: It’s not just family; it’s also all the cool things around us.
• I haven’t gone through the Hadoop Summit archives yet, but it looks as if there’s a lot of insight there about current Hadoop application activity.
• If you’re a cat lover and don’t hate simple/traditional music, check out Marc Gunn’s cat filksongs, especially the infectious “What Shall We Do With a Catnipped Kitty?” and “Lord of the Pounce”, both playable from the right sidebar of that page (#7 and #10 respectively). Gunn is also a chief perpetrator of the justly (in)famous Do Virgins Taste Better? cycle of filksongs.
• Former SAP exec Dennis Moore offers a theory as to why SAP cares so much about in-memory DBMS. It’s to integrate business processes, because SAP has no other software layer good at doing same. Interestingly, Dennis originated SAP’s previous attempt at meeting a similar need via its composite applications initiative. However, in Dennis’ view this benefit would only be achieved by a major rewrite of SAP’s applications.

And it’s not going to get any better. Next week = the oft-postponed elder care trip. Then I’m back for a short week. Then I’m off on my quarterly visit to the SF area. Soon thereafter I’ve have a lot to do in connection with Enzee Universe. And at that point another month will have gone by.

Anyhow:

• Back in January, Oracle finally briefed me on Exadata 2. I also requested and got permission to post what I regarded as pretty interesting slides, then never got around to doing so. Well, here they are. (Pay no attention to the word “Confidential”.)
• Two people I have a lot of respect for, Cindi Howson and Doug Henschen, seem bullish on SAP’s in-memory NewDB efforts. But for a variety of execution reasons, I’m skeptical that this will matter for anything except SAP’s analytics suite. I.e., I don’t think anybody much except SAP will write OLTP apps to it, and I don’t think that without OLTP apps being written to it it’s much more than Business Objects’ answer to QlikView.
• I just learned that Netezza’s previous geospatial technology didn’t get ported to TwinFin. However, Netezza obviously found a geospatial alternative.

I ‘m beginning to make a habit of asking vendors for a postable version of their slide decks. Sybase IQ is another example.

• Google is doing something called BigQuery that is “SQL-like” for big data analytics. I don’t know anything about it.
• I also don’t know anything about IBM BigSheets yet. It sounds something like Datameer, but that could be way off the mark.

That said,  

• Raj said that Sybase’s Aleri acquisition was, if anything, tracking ahead of expectations.
• Raj didn’t seem the slightest bit focused on the Coral8/Aleri CEP-based BI strategy that John Morell had long championed.
• Raj reminded me that Sybase SQL Anywhere has numerous OEMs, not just on the true desktop/laptop or smaller, but also in a return to its server/workgroup roots. Sybase SQL Anywhere even added geospatial indexing recently.

Raj also spoke glowingly of SAP’s in-memory database technology and the potential for Sybase of same — until I asked a follow-up question. At that point, he confessed that he didn’t really know much about about SAP’s in-memory database technology yet. As I said before, I believe SAP is fairly sincere about its belief that its in-memory database technology will conquer the world — but this is a naive and poorly-founded opinion even so.

One tidbit I did get is that SAP’s in-memory database technology is not just son-of-T-REX. A Korean (Raj thinks) company SAP had acquired is also in the mix. Raj also had the impression SAP’s in-memory technology can do rows, columns, or hybrid structures. On the one hand, that makes sense. On the other, it’s not a perfect fit with what Hasso Plattner said last year.

4 1/2 years ago, I noted that SAP believed strongly in database proliferation:

2. One big benefit they see to this strategy is that it reduces the need to have grand integrated databases. If one application manages data for an entity that is also important to another application, the two applications can exchange messages about the entity. Anyhow, many of their comments make it clear that, between partner company databases (a bit of a future) and legacy app databases (a very big factor in the present day), SAP is constantly aware of situations in which a single integrated database in infeasible.
…
4. One area where SAP definitely favors redundancy and synchronization is data warehousing. Indeed, they have an ever more elaborate staging system to move data from operational to analytic systems.

Per SAP CTO Vishal Sikka, quoted in Merv Adrian’s blog, they still do:

Last week I was at one of our largest consumer goods customers. They have 3000 data sources worldwide, and coordinating across all of them is a formidable challenge. Imagine our in-memory database sitting next to them, in hundreds of departments. To make decisions you need the data in synch in real time. We can use Sybase’s Replication Server, and event technology, to transform what companies can aspire to do.”

And by the way, MaxDB’s significant share in the SAP user base is evidence as well. (I expect that share to go to Sybase as the decade progresses.)

This all also fits well with Sybase IQ’s usage model.