April 7, 2012

Many kinds of memory-centric data management

I’m frequently asked to generalize in some way about in-memory or memory-centric data management. I can start:

Getting more specific than that is hard, however, because:

Consider, for example, some of the in-memory data management ideas kicking around.

And that, kiddies, is why I hesitate to generalize in too much detail about “in-memory database management.”

Despite its length, this is still a very partial list of memory-centric data management approaches. I encourage you to add other examples into the comments that I might have left out.

Related link

Comments

14 Responses to “Many kinds of memory-centric data management”

  1. Brian Andersen on April 7th, 2012 10:26 pm

    Really great overview of the approaches used in various products. An approach that I have seen used with success in (adhoc) applications is to keep the raw facts on disk and to keep pre-aggregated datasets at various levels of granularity in memory. This allows “human real-time” slicing and dicing of very large datasets, because a lot of the grunt work has already been done. I am a little surprised that commercial products have not incorporated this idea.

  2. Ian in London on April 8th, 2012 5:12 am

    It would be interesting to read Kurt review Microsoft’s Analysis Services Tabular Mode in-memory offering.

  3. Jean-Michel Franco on April 8th, 2012 7:02 am

    Very interesting post.
    Regarding what you mention about Timesten, my understanding is that Oracle Exalytics comes with a columnar version of Timesten, making it more similar to SAP Hana except that I’m unsure you can have columnar and row based into one single DB instance. Also I’m unsure Timesten has write capability in its columnar version ; for that, Exalytics comes with an in-memory version of Essbase (however, given Essbase architecture, I’m unsure that it can provide sparse MOLAP).

    As Ian, I would be interested to have your POV around xVelocity from Microsoft. Also, although I know you consider Actian Vectorwise with some caution, I would be intersesting in how you would position the way it uses CPU Cache as execution memory. Would you consider it as in-memory too ?

  4. Curt Monash on April 8th, 2012 3:07 pm

    Jean-Michel,

    You’ve listed a few very frustrating subjects.

    First, Oracle likes to imply it has columnar storage when it really just has columnar compression. I’ve heard nothing about TimesTen being an exception.

    Second, Ingres has lied rather badly about the contents of my previous conversations with them, in an attempt to discredit my opinion of them. That puts them on a very short blacklist of companies I will not risk talking to, for fear of adding further fuel to the slanderous fire. (Basho is the other big example.)

    Third, Microsoft is not in the habit of briefing me if they can help it. They evidently prefer analysts who are easier to control. Oracle takes a similar approach, but not to the same extreme.

  5. Michelle Agul on April 8th, 2012 6:43 pm

    Jean-Michel,

    Curt is absolutely correct regarding the new Oracle TimesTen(v.11.2.2)specifically for Exalytics. It includes analytic functions along with in-memory columnar compression (~5x). The compression is specified against individual columns or groups of columns. Although compressed columns take longer to load, they will theoretically be much faster to query (than uncompressed columns), and obviously, take up less disk space.

    The idea is to load aggregates into TimesTen, leave base data at source, and use OBIEE’s vertical federation capability to seamlessly report across both (all transparent to the end-user).

    Mark Rittman, of UK-based Rittman Mead Consulting, did a terrific job back in late February http://bit.ly/HtFEWo providing a much deeper dive than Oracle does in any of its white papers for public viewing.

  6. Curt Monash on April 8th, 2012 8:45 pm

    Edit: Never mind — I misread Michelle’s comment.

    Michelle,

    I just drew a distinction between columnar storage and columnar compression, and now you’re commenting as if they’re the same thing. :(

  7. Curt Monash on April 8th, 2012 8:47 pm

    And Mark’s link doesn’t say anything about TimesTen having columnar storage.

  8. Harri Kallio on April 9th, 2012 5:59 am

    An SAP company Sybase has also in-memory database in ASE product.
    http://www.sybase.com/manage/in-memory-databases

    Which is also used in Sybase RAP product as RAP Cache.

    Nevertheless of personal Actian/Ingres relationships, in theory-wise interesting paper about pre-VectorWise DBMS MonetDB breaking the “memory wall”:
    http://oai.cwi.nl/oai/asset/13805/13805B.pdf

  9. Keshav Murthy on April 9th, 2012 3:07 pm

    IBM Informix has an in-memory engine, Informix Warehouse accelerator(IWA). It uses hybrid columnar approach & is targeted for data analysis. The product details are at http://ibm.co/f0Hc2h and http://ibm.co/xQxaD2. IWA takes snapshot of the data, compresses it and runs queries on compressed data, without decompression when possible. For details, see the references at the end of the paper http://ibm.co/xQxaD2.

  10. Darpan Dinker on April 24th, 2012 7:08 pm

    With respect to “real-time analytics” and hence maintaining data in DRAM… it may be interesting to compare the rate of data growth and Moore’s law.

  11. Curt Monash on April 24th, 2012 8:13 pm

    Hence my point that some kinds of data are clearly destined to be held entirely in RAM, while other kinds will rely on cheaper storage for a long time to come.

  12. Disk, flash, and RAM | DBMS 2 : DataBase Management System Services on July 12th, 2012 11:08 pm

    [...] months ago, I pointed out that it is hard to generalize about memory-centric database management, because there are so many different kinds. That said, there are some basic points that I’d [...]

  13. In-memory, (hybrid) memory-centric DBMS — three analytic glossary draft entries | DBMS 2 : DataBase Management System Services on August 20th, 2012 2:31 pm

    [...] Many examples of memory-centric data management (April, 2012) [...]

  14. Notes on memory-centric data management | DBMS 2 : DataBase Management System Services on January 3rd, 2014 8:12 am

    [...] The broad variety of memory-centric data management approaches. [...]

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.