Thoughts on in-memory columnar add-ons
Oracle announced its in-memory columnar option Sunday. As usual, I wasn’t briefed; still, I have some observations. For starters:
- Oracle, IBM (Edit: See the rebuttal comment below), and Microsoft are all doing something similar …
- … because it makes sense.
- The basic idea is to take the technology that manages indexes — which are basically columns+pointers — and massage it into an actual column store. However …
- … the devil is in the details. See, for example, my May post on IBM’s version, called BLU, outlining all the engineering IBM did around that feature.
- Notwithstanding certain merits of this approach, I don’t believe in complete alternatives to analytic RDBMS. The rise of analytic DBMS oriented toward multi-structured data just strengthens that point.
I’d also add that Larry Ellison’s pitch “build columns to avoid all that index messiness” sounds like 80% bunk. The physical overhead should be at least as bad, and the main saving in administrative overhead should be that, in effect, you’re indexing ALL columns rather than picking and choosing.
Anyhow, this technology should be viewed as applying to traditional business transaction data, much more than to — for example — web interaction logs, or other machine-generated data. My thoughts around that distinction start:
- I argued back in 2011 that traditional databases will wind up in RAM, basically because …
- … Moore’s Law will make it ever cheaper to store them there.
- Still, cheaper != cheap, so this is a technology only to use with your most valuable data — i.e., that transactional stuff.
- These are very tabular technologies, without much in the way of multi-structured data support.
But in a bit of evidence that disconfirms my case, one of the first SAP applications to require HANA was something called “Smart Meter Analytics”.
To see more specifically where this technology could be useful, let’s map it against my 2011 analytic database taxonomy.
- If you’re managing a partial EDW (Enterprise Data Warehouse) on the same technology as your OLTP (OnLine Transaction Processing) databases, but are running out of steam, in-memory columnar could provide some acceleration.
- Traditional data marts are somewhat obsolete, and establishing a new one would be mainly a cost play. So the fit is questionable.
- Investigative data marts could be a good fit, but only if you’re fairly unimaginative as to the kinds of data you want to include.
- Several other categories are no fit at all.
- There’s a good fit for certain kinds of operational analytics.
I’ll finish by expanding on that last point.
Operational applications have always had analytics blended in. If nothing else, there were a lot of straight reports; sometimes there’s a bit of optimization as well. Workday, for example, has BI and search as two of its core OLTP UI metaphors, and has a lot of other BI snippets called worklets as well. (And by the way, a lot of Workday’s database is in-memory.) I’ve thought for years that operational/analytic blending would be a major area of competition between Oracle and SAP; hence — I believe — SAP’s acquisitions of Business Objects and KXEN. Columnar in-memory Oracle features, and similarly SAP HANA, seem well-suited to support such application elements.
Comments
6 Responses to “Thoughts on in-memory columnar add-ons”
Leave a Reply
Will be interesting to see the details (whitepapers). I have yet to read the BLU papers but with what little I know I think Microsoft took a very different approach than Oracle and I like the Hekaton effort — both the technology and integration.
[…] To recap the OpenWorld news: Ellison kicked off it all off Sunday night with news of an in-memory database option for the latest Oracle 12C database which he said will run at “ungodly speed.” The in-memory option — which competes with the aforementioned HANA — will be broadly available next year. (Check out database guru Curt Monash’s take on the in-memory database fervor here.) […]
[…] Option. While there is already a great discussion at Rob’s blog and further analysis at Curt’s, I thought I could add a […]
Curt,
In this post you imply that BLU leverages the technology that manages indexes and massages it into an actual column store. That’s far from true. (Afaict it is true in the case of SQL Server’s Columnstore indexes, while I don’t know what 12c will be doing.)
For more information about BLU, I recommend you read our very detailed VLDB 2013 paper, where we thoroughly describe (in uncharacteristic for an industrial track research paper detail) how BLU operates: https://t.co/KuayqEXqyG
Best regards,
-Ippokratis.
Ippokratis,
Thanks! Edited accordingly.
[…] Thoughts on in-memory columnar add-ons (dbms2.com) […]