• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays.
• Human/Nontabular data — i.e., all other data generated by humans.
• Machine-Generated data.

I won’t now stand by every nuance in that post, which may differ slightly from those in my more recent posts about machine-generated data and poly-structured databases. But one general idea is hard to dispute:

Traditional database data — records of human transactional activity, referred to as “Human/Tabular data above” — will not grow as fast as Moore’s Law makes computer chips cheaper.

And that point has a straightforward corollary, namely:

It will become ever more affordable to put traditional database data entirely into RAM. 

Actually, there are numerous ways for OLTP, other short-request, and some analytic databases to wind up in RAM.

• SAP has some good ideas for how it could happen, banging transactions into what is essentially an in-memory analytic database. (I dispute SAP’s claims of transformational database technology leadership, but that doesn’t mean the underlying ideas aren’t good.)
• For those who can afford the associated technology disruption, memory-centric object-oriented DBMS could be appealing.
• Web scalability best practices commonly include keeping data in RAM (e.g., that’s pretty much the point of caching layer memcached).
• SaaS (Software as a Service) companies — such as Workday — often bring a particular tenant’s database entirely into RAM.
• QlikView highlights the benefits of doing business intelligence in RAM.
• SAS HPA makes the argument that even “big data analytics” should sometimes be done in RAM.
• I don’t have particularly favorable opinions at this time about marketing strategies or momentum at Oracle TimesTen, IBM solidDB, or VoltDB, but those examples at least serve to illustrate that memory-centric OLTP DBMS have existed for years.
• Actually, SAP has at least two good ideas, if you count Sybase as part of SAP.

And here’s the kicker: Intel told me last year that CPUs are headed to 46-bit address spaces around mid-decade. Indeed, they hired me to help figure out if that was enough.* That multiplies out to 64 terabytes of RAM on a single server, chip costs permitting. So most of what we now think of as operational databases — and many of the analytic ones too — will fit in-memory, even if they run very large businesses.

*And did so without putting the discussion under any kind of NDA.

Likely consequences of all this include:

• Legacy apps will (eventually) be consolidated and virtualized in-memory. Their underlying databases will grow so slowly that eventually the cost of putting them in RAM will be too low to worry about.
• Expensive storage systems will (continue to) be irrelevant to database processing. Databases that don’t fit in RAM will typically be big enough to require the attention of a lot of CPUs — and in those cases the DBMS software itself will handle all the storage tasks.
• Major OLTP DBMS vendors, such as Oracle, will need alternate in-memory code lines, because disk-centric architectures are sub-optimal in-memory. Well, that’s what they have those big R&D budgets for.
• SaaS vendors and web businesses may not rely on today’s major OLTP DBMS vendors. (I was going to say “won’t” rather than “may not” until I recalled the likely M&A endgame.) Traditional enterprises may blanch at migrating away from their legacy DBMS environments, but the trade-offs are different for technology companies using DBMS as subsystems.

Of course, the same trends that make data-storing chips cheaper will make data-generating chips cheaper too. So, just as there are huge amounts of machine-generated data that you’d never pay to store in RAM, the same will still be true 10 years from now; the data volumes involved will just be a lot bigger. And thus there will still be plenty of very large analytic databases using relatively cheap forms of storage, perhaps even disk.

But OLTP and other short-request processing are likely to wind up in-memory. And the same may be true for a considerable amount of analytics, especially but not only if the analytics have a low-latency requirement.

The Oracle Database 11g Release 2 white paper I cited a couple of weeks ago has evidently been edited, given that a phrase I quoted last month is no longer to be found. Anyhow, here are some quotes from and comments on what evidently is the latest version.

The In-Memory Database Cache (IMDB Cache) option of Oracle Database 11g Release 2, allows data to be cached and processed in the memory of the applications themselves, off-loading the data processing to middle tier resources. Any network latency between the middle tier and the back-end database is removed from the transaction path, with the result that individual transactions can often be executed up to 10 times faster. This is particularly useful where very high rates of transaction processing is required, such as those found under market trading systems, Telco switching systems, and Real Time manufacturing environments. All data in the middle tier is fully protected through local recovery, and asynchronous posting to the back end Oracle Database. With Oracle Database 11g Release 2, the ability to transparently deploy IMDB Cache with existing Oracle applications becomes much easier – with common data types, SQL and PL/SQL support, and native support for the Oracle Call Interface (OCI).

At a guess, this sounds like it’s based on Oracle’s TimesTen acquisition.

Oracle Database 11g Release 2 adds further optimizations, including capabilities to automatically determine the most optimal degree of parallelization for a query, based on available resources. With this comes automated parallel statement queuing, where the database determines that, based on current resource availability, it is more effective to queue a query for later execution once required resources have freed up.

Sounds like a kind of automatic workload management — i.e., the kind of optimization vendors of mature products get around to putting into their systems. It does not sound like query pipelining, however.

Oracle Database 11g Release 2 will automatically distribute a large compressed table (or a smaller non-compressed table), into the available memory across all the servers in the Grid, and will then localize parallel query processing to the data in memory on the individual nodes. This dramatically improves query performance, and is especially useful where large tables can be entirely compressed into the available memory using compression capabilities.

So Oracle caches compressed data. Not stated is which compression techniques are covered.

Each Exadata Storage Server stores up to 7 Terabyte [sic] of uncompressed user data, and also comes enabled with 384 GB of solid-state Flash cache. This Flash Cache automatically caches active data of the magnetic disks in the Oracle Exadata Storage Server, delivering a 10x performance gain for read and write operations under OLTP applications.

Sounds like the Flash memory is positioned for OLTP use.

In the past, Database Administrators and System Administrators have spent a great deal of time determining to how best place data across these disk arrays, to get maximum performance and availability. The best procedure for data placement is to simply Stripe And Mirror Everything; stripe data blocks equally across all disks in an array, and then mirror the blocks on at least two disks. This approach provides the perfect balance between performance, disk utilization, and ease of use.

This is a big part of what could be called the “Administering Oracle doesn’t suck nearly as badly as it used to” pitch. (Mitchell Kertzman, who was Sybase CEO after the mid-1990s meltdown, told me his motto was “We suck less every day.” But I digress …)

Automatic Storage Management (ASM), a feature of Oracle Database 11g automates the striping and mirroring of database without the need to purchase third party volume management software. As data volumes increase, additional disks can be added, and ASM will automatically restripe and rebalance the data across available disks to ensure optimal performance. Similarly, disks that report errors can be removed from the disk array, and ASM will re-adjust accordingly.

I.e., you can add nodes without taking the system down. That’s becoming a pretty standard feature for serious parallel DBMS.

Oracle Database 11g Release 2 improves ASM in significant areas. New intelligent data placement capabilities store infrequently accessed data on the inner rings of the physical disks, while frequently accessed data is placed on the outer rings, offering better performance optimization.

Also pretty standard.

Oracle has been enhancing partitioning capabilities for over ten years. Oracle Partitioning, an option of Oracle Database 11g Release 2, allows very large tables (and their associated indexes) to be partitioned into smaller, more manageable units, providing a “divide and conquer” approach to very large database management. Partitioning also improves performance, as the optimizer will prune queries to only use the relevant partitions of a table or index in a lookup. Oracle Database 11g Release 2 provides multiple methods for partitioning data, and also allows different levels of partitioning on the same table, so that a single partitioning strategy can be used to improve both performance and manageability.

Even better might be a system that doesn’t lean heavily on complex partitioning to achieve good performance.

Oracle Partitioning can also manage the lifecycle of information. Typically, all databases have active data – the information being processed this month or quarter, and historical data that is primarily read-only. Organizations can take advantage of the inherent lifecycle of data to implement a multi-tiered storage solution and lower their overall storage costs. For example, a large table within an order-entry system could contain all the orders processed in the last 7 years. Oracle Partitioning can be used to set up monthly partitions, with the current last four months of order data partitioned onto a high-end storage array, with all the other partitions placed on a lower-cost storage solution, often 2-3 times less cost than the high end storage environment.

This is becoming a standard feature for any parallel DBMS that can support multiple kinds of storage in one system.

Oracle Database 11g also provides advanced compression techniques to further reduce storage requirements. Using Oracle Advanced Compression, an option to Oracle Database 11g, all data in a table can be compressed using a continuous table compression capability that achieves a 2-4 times compression ratio with little performance impact on OLTP or Data Warehousing workloads. This compression technology replaces duplicate values in a table with a single value, and continuously adapts to data changes over time, so compression ratios are always maintained.

Sounds like dictionary/token compression.

With Oracle Database 11g Release 2, the Exadata Storage Servers in the Sun Oracle Database Machine also enable new hybrid columnar compression technology that provides up to a 10 times compression ratio, with corresponding improvements in query performance. And, for pure historical data, a new archival level of hybrid columnar compression can be used that provides up to 50 times compression ratios.

I thought they said 40X before. But even if my memory isn’t playing tricks regarding that, single-point compression ratio estimates are always very approximate.

Any hardware component in an Oracle Grid can be dynamically added or removed as required. Disks can be added or removed online with ASM, with the data automatically rebalanced across the new disk infrastructure. Additional servers can also be easily added or removed to a Real Application Cluster with users connected to these nodes rebalanced across the infrastructure. This ability to migrate users from one server to another in a RAC cluster also enables rolling patching of the database software. If a patch needs to be applied, then a server can be removed from the cluster, patched, and then put back into the cluster. The same operation can be repeated for the next server in the cluster, and so on.

Nice. And the paper goes on in that vein for quite a while.

Oracle Total Recall, an option to Oracle Database 11g, provides a solution for the retention of historical information. With Oracle Total Recall, all changes made to data are kept to provide a complete change history of information. This means that auditors can not only see who did what when, but they can also see what the actual information was at the time – something that previously has only be [sic] available by building into the application, or by expensive backup retention policies.

Timestamping/time-travel/whatever is increasingly becoming a standard feature as well, especially given the number of PostgreSQL-based DBMS on the market.

New internal control requirements found in regulations can be difficult and expensive to implement in an environment with multiple applications. Oracle Database Vault, an option to Oracle Database 11g, allows access controls to be transparently applied underneath existing applications. Users can be prevented from accessing specific application data, or from accessing the database outside of normal hours; separation-of-duty requirements can be enforced for different Database Administrators without a costly least privilege exercise. And Oracle Advanced Security, an option to Oracle Database 11g, can be used to transparently encrypt data at all levels – data in transit on the network; data at rest on physical storage and in backups. Similarly, the Data Masking pack can be used to obfuscate data as it moves from production to development, reducing the potential violation of privacy regulations or risking sensitive data leaks.

Oracle is the gold standard in database security.

Oracle’s self-management approach takes two tacks. Firstly, wherever possible, repeatable, labor intensive and error prone tasks that can be fully automated in the database have been. For example, Storage Management, Memory Management, Statistics collection, Backup and Recovery, and SQL Tuning have all been automated. Secondly, where operations cannot be fully automated, intelligent advisors are built into the database to mentor Database Administrators on how to get the best out of their systems. Advisors are provided for Configuration Management, Patching, Indexing, Partitioning, Performance Diagnostics, Data Recovery, and, new in Oracle Database 11g Release 2, Compression and Maximum Availability.

And boy are they needed.

Recent studies performed by an independent research company shows that Database Administrators can expect to spend 26% less time managing their 11g environments over their 10g environments, and as much as 50% when compared to older Oracle9i deployments.

50% of way too much is still way too much.

… a real-time inventory control system which has the following requirements — basically it needs to provide high write-through-rate, and it needs little if any indexing functionality.

1) a central control system records/updates the inventory data (number/weight and etc.) at each room/rack — there exist thousands of racks/rooms

2) sensors (at different rack) also report/update the temperature to the central control system, at rate of approximately 1~2 updates/min — as there are thousands of sensors, the update throughput needs to be high considering the scalability requirement.

3) When some problems happen, we need to roll back the logs (to replay the events and diagnose the root cause).

His questions included:

Memory-centric DBMS or complex event/stream processing (CEP)? Given that the purpose is to record data, and that he wants to record all data rather than engage in immediate data reduction, true DBMS seems like the way to go.

What are the good in-memory DBMS alternatives anyway? He thought McObject’s eXtremeDB was dominant in the market, which surprised me (although McObject does seem to have made a push in Asia). I know long-time leaders TimesTen and solidDB have, since their acquisitions by Oracle and IBM respectively, pulled back from the standalone market. (Their new owners are more interested in front-end caching for Oracle and DB2 respectively.) But I didn’t think the pullback had been that — as it were — extreme.

And while my correspondent didn’t ask this, I’ll add — should he maybe just go with an ordinary DBMS anyway? A couple thousand updates per minute isn’t that forbidding. On the other hand, it might be hard to achieve with conventional DBMS on super-cheap hardware. And a memory-centric alternative that only logs to disk in near-real-time might be plenty good enough for any analytics they want to do.

He was quite frank about wanting to get experience with leading-edge technology, with an eye to deploying it in other use cases.  So it’s reasonable to be pretty general in this whole discussion. With that as background — well, I’ve already given some of my thoughts.  So what are yours?

• Congratulations to Bill Janeway, Cary Davis and the rest of the Warburg Pincus team. They started BEA with a buyout of the ancient Tuxedo TP monitor. They aggressively bought Weblogic. They navigated the many changes in the core definition of the app server market. They recovered from the stumble in app dev tools. Yes, management ultimately is responsible, but great investors and board members had a lot to do with this one.
• TIBCO is an obvious candidate to go next, with all the major elements — significant middleware, unclear business model going forward, plus some minor sexy acquisitions (e.g., Spotfire) and internal efforts (e.g., in CEP) anybody would like to own.
• Bex Huff says BEA has a “lock” on middleware for the financial services industry.
• Tim Bass notes that Oracle picks up a CEP product in the deal. In general, Oracle is potentially a major threat in the CEP business. High-end transaction processing is what it does. And TimesTen’s core market is directly adjacent to CEP.
• Ray Wang notes that BEA has a big presence in China. He also thinks SAP’s NetWeaver is in trouble.
• The three obvious buyers for SAP are Microsoft, HP, and IBM, probably in that order.

My prior coverage of the Oracle/BEA merger includes a brag about having called it back in 2002, a quick analysis of how Oracle should reconcile its various products, and an obvious joke.

• solidDB is actually a very interesting hybrid disk/in-memory memory-centric database management system. However, the press release announcing the deal makes it sound as if solidDB is in-memory only.
• That strongly suggests that IBM is buying Solid mainly to compete with Oracle TimesTen. As of last June, solidDB was already IBM’s TimesTen answer via a partnership; this deal just solidifies that arrangement.
• This probably isn’t good news for Solid’s MySQL engine. That’s a pity, since solidDB technically has the potential to be the best MySQL engine around.
• Notwithstanding IBM’s presumed intentions, Solid’s main market success historically is as an embedded system in telecommunications equipment, network software, and similar systems.
• Last year I wrote a white paper on memory-centric data management, showcasing four products. IBM now has bought two of them, namely Solid’s and Applix’s (via Cognos).
• Comparisons to IBM’s embedded Java DBMS Cloudscape are pointless. That’s just a failed product vs. solidDB or Sybase SQL Anywhere, and IBM long ago cut its losses.

Here’s a link to a webcast I did with Solid back in June.

Edit: As per the invitations to the 11:00 am conference call, the following — straight from the press release — is evidently the official IBM one-paragraph description of the acquisition.

The acquisition of Solid supports IBM’s global Information on Demand strategy by adding real-time data access capabilities to the company’s portfolio of database and information management offerings. Solid’s software uses in-memory database technology to quickly retrieve data from a computer’s memory (or RAM). Using this software, businesses can access and store data at speeds up to ten times faster than traditional disk database systems. Solid’s architecture also enables applications to recover from system failure rapidly and to require little disk space and almost no hands-on administration.

Edit: IDG has quotes confirming the emphasis on TimesTen competition. IDG also reminds us that MySQL 6.0 will have its own storage engine Falcon. I guess that’s pretty much do or die for MySQL now.

Analogies to Oracle/TimesTen are probably not coincidental; this is exactly the upside scenario for the TimesTen acquisition, as well as being TimesTen’s biggest growth area towards the end of its stint as an independent company.

• Object caching (e.g., Tangosol, Progress ObjectStore)
• In-memory RDBMS (e.g., Oracle TimesTen, Solid BoostEngine, McObject eXtremeDB)
• Stream processing (e.g., Progress Apama, Streambase)

And that’s just for OLTP; there’s a whole other set of memory-centric technologies for analytics as well.

When one connects the dots, I think three major points jump out:

1. There’s a lot more to high-end OLTP than relational database management.
2. Oracle is determined to be the leader in as many of those areas as possible.
3. This all fits the market disruption narrative.

I write about Point #1 all the time. So this time around let me expand a little more on #2 and #3.

Except at the high end, banging OLTP data into a relational database is pretty much a commodity. You need transaction integrity, referential integrity, 24×7 operation – and what else? Answer that “what else”, and you’ll probably be talking about scalability, TCO, or both.*

*Well, there’s also unattended/embedded operation. But – and here I’m putting the point very mildly – that’s hardly an area of Oracle competitive superiority.

One Oracle response is to provide lots of add-on technologies for high-end customers, on the database and middle tiers alike. In app servers it’s done surprisingly well against BEA. It’s sold a lot of clustering. And it’s bought into and tried to popularize niche technologies like TimesTen and Tangosol’s.

This all makes perfect sense – it’s a great fit for Oracle’s best customers, and a way to get thousands of extra dollars per server from enterprises that may already have bought all-you-can-eat licenses to the Oracle DBMS. And being so sensible, it fits into the Clayton Christensen disruption story in two ways:

1. Oracle may be helpless against mid-tier competition, but it sure has the high-end core of its market locked up.
2. As one type of technology is commoditized, value is created in other parts of the technology stack.

The latter point is particularly important, because it applies not just to the new high-end systems software, but also to the whole application/analytics focus of Oracle’s much bigger strategic moves. And it’s one I believe in. Software features have been going from niche to mainstream to commodity, and being replaced by newer compatriots, for as long as there’s been a software industry.

• Oracle lags behind both IBM and Microsoft on native XML. I can’t recall Oracle trailing both those vendors at once on anything as major before.
• I’ve come to believe that memory-centric technology is a really powerful improvement to DBMS. Oracle’s done nothing effective inhouse in this area. Buying TimesTen was a good move, however.
• I’m growing less inclined to excuse the ongoing complexity of Oracle database administration. From the classic Progress DBMS to Solid’s truly zero-DBA system (embed it in telecom equipment without a keyboard or monitor, and it runs for years) to ANTs’ new challenge, there are a whole lot of fairly full-featured RDBMS with a whole lot less knobs and dials. Do they match Oracle’s full functionality? No. But could they handle a lot of the apps which run on Oracle today? In some cases, quite possibly. So why hasn’t Oracle itself come up with a plug-compatible Oracle Lite that truly is lite?
• High-end data warehouse scaling continues to be a problem for Oracle. Yes, the effective upper bound on Oracle database size keeps increasing. But so does the size of the largest data warehouses, and the latter number is definitely greater than the former.
• Microsoft Analysis Services is quite cool. Why hasn’t Oracle done the same thing?
• There are many seemingly major feature advantages that Oracle has failed to exploit. Robust text processing in the DBMS and Virtual Private Database (a cool security feature) are two of the biggest that come to mind. They’ve sold well to customers who were predisposed to want them, but Oracle hasn’t had the stomach to broaden the market for them. Yes, that’s more of a marketing problem than a technical one, but after a decade of squandering your technical advantages, you can find that your people lose the enthusiasm to create more of them. What else has Oracle done in years that is as functionally innovative? Nothing comes to mind.
• The whole apps-vendor-consolidation strategy is often presented along with a huge white flag regarding DBMS market share growth.
• Oracle has never, ever solved its applications product problems. Ditto its BI development problems. Both these fiascos are plausibility arguments for the thesis that the company can NOT reverse bad product philosophy choices, no matter how much market evidence slaps them in the face, and no matter how adaptable the company is in other ways.
• Oracle’s analyst relations have gotten very defensive and insular. To some extent, that happens at most big companies. But I cannot recall a previous time when IBM’s and Microsoft’s DBMS operations were both more accessible than Oracle’s, and I’ve been a DBMS analyst since the industry’s very early days. Often when a company acts this bizarrely, it does so because it has a great deal to hide.

That’s a lot of evidence, even without mentioning threats from the open sourcers and the data warehouse appliance guys. So why am I not wholly convinced yet? Well, reasons include a variety of scalability features, extensibility features that are rivaled only by IBM’s, market share dominance on Linux, and Andy Mendelsohn. That’s a pretty compelling list too. Still, the Oracle colossus is teetering a little bit, and it’s not beyond imagination that some future earthquake could bring it crashing down.

I’ll give a quick summary of the vendors and products I am focusing on in this newly-named category, and it should be clearer what I mean:

• TimesTen (now owned by Oracle): TimesTen is the quintessentional “in-memory DBMS.” It’s a fairly full relational DBMS, but if you want to persist memory to disk it has to be handed off to a conventional DBMS. Historically, that has usually been MySQL or Oracle. TimesTen’s biggest market penetration has been in financial trading.
• Solid Information Technology‘s BoostEngine: Solid is a Finnish company (or was — it’s pretty American now) specializing in embedded DBMS sold mainly for telecommunication uses. Big OEM customers include several well-known telecom equipment manufacturers and HP (for OpenView). “Embedded” often means no DBA, no monitor, no keyboard — they box manufacturer installs it and there it stays for the life of the product. Solid has to offer strong replication capabilities, since its products are often used in highly distributed (e.g., multiblade, multibox) environments. So it’s taken the next step and exploited the replication by allowing customers to use some instances of the product disklessly.
• Event-stream products from Streambase and Progress: The canonical application for event-stream products is automating financial trading decisions based on the flow of market information. Mike Stonebraker, the brains behind Streambase, has recently popularized the idea; Progress bought Apama, who actually have been in the business longer. These applications require even more speed than the financial trading apps that TimesTen handles, and they discard most of the information they look at. In-memory is the only way to go.
• Progress’s ObjectStore: ObjectStore comes from the company Object Design, which merged into Excelon, which was acquired by Progress. It’s really a toolkit for building DBMS and similar systems, which is why it’s at various times been marketed as an OODBMS and an XML DBMS, without a lot of success either way. But there have been a few sterling apps built in ObjectStore even so, including a key part of the Amazon bookstore Despite this limited market success, a significant fraction of Progress’s best engineering talent has moved over to the Real-Time Division to focus on ObjectStore and other memory-centric products. The memory-centric aspect of ObjectStore is this: ObjectStore’s big virtue is that it gets objects from disk to memory and vice-versa very efficiently, then distributes and caches them around a network as needed. This was originally invented for client/server processing, but works fine in a multi-server thin client setup as well. And object processing, of course, relies on a whole lot of pointers. And pointer-chasing is pretty much the worst way to deal with the disk speed barrier, unless you do it in main memory.
• Applix‘s TM1: Like many companies in the analytics area, Applix has had trouble deciding whether it sells applications, BI system software, or both. But in any case its core technology is TM1, a memory-centric MOLAP offering. Traditional MOLAP products reside on the horns of a nasty dilemma: They rely on precalculation to give good performance, but that causes ghastly database explosion. Applix gets out of this problem by doing no precalculation whatsoever, loading the data into main memory, and executing all queries on the fly.
• SAP’s BI Accelerator: SAP is building out an elaborate technology stack with NetWeaver, especially in the BI area. One important aspect is that the full data warehouse is logically broken (or copied) into a series of data marts called “InfoCubes.” BI Accelerator takes the logical next step, loading an entire InfoCube into main memory. Almost every query is executed via a full table scan, which would be insane on disk but makes perfect sense when the data is already in RAM.

So there you have it. There are a whole lot of technologies out there that manage data in RAM, in ways that would make little or no sense if disks were more intimately involved. Conventional DBMS also try to exploit RAM and limit disk access, via caching; but generally the data access methods they use in RAM are pretty similar to those they use when going out to disk. So memory-centric systems can have a major advantage.