*I could be wrong in those guesses; as small companies go, McObject is unusually prone to secrecy games.

As best I understand:

• eXtremeDB is something like an in-memory object-oriented DBMS, designed to be embeddable.
• However, much as with Objectivity and other old-school OODBMS, eXtremeDB winds up being more of a toolkit with which to build DBMS than a full DBMS.
• eXtremeDB has a few indexing schemes. The main one is good old B-trees. One customer wanted Patricia tries, so they’re in there. (Perhaps not coincidentally, solidDB relies on Patricia tries.) At least one wanted R-trees, so they’re in there too.
• eXtremeDB has long had the option of persistent logs.
• eXtremeDB newly has a hybrid memory-centric option, in which you can have more data in the database than fits into RAM.
• eXtremeDB newly has multi-master two-phase-commit clustering.

My guess three years ago that eXtremeDB might emerge as an alternative to solidDB seems to have been borne out. McObject CEO Steve Graves says that the core of McObject’s business is OEMs, in sectors such as telecom equipment and defense/aerospace. That’s exactly solidDB’s traditional market, except that solidDB got acquired by IBM and deemphasized it.

I’ve said before that if I were starting a SaaS effort — and it wasn’t just focused on analytics — I’d look at using a memory-centric OODBMS. Perhaps eXtremeDB is worth looking at in such scenarios.

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays.
• Human/Nontabular data — i.e., all other data generated by humans.
• Machine-Generated data.

I won’t now stand by every nuance in that post, which may differ slightly from those in my more recent posts about machine-generated data and poly-structured databases. But one general idea is hard to dispute:

Traditional database data — records of human transactional activity, referred to as “Human/Tabular data above” — will not grow as fast as Moore’s Law makes computer chips cheaper.

And that point has a straightforward corollary, namely:

It will become ever more affordable to put traditional database data entirely into RAM. 

Actually, there are numerous ways for OLTP, other short-request, and some analytic databases to wind up in RAM.

• SAP has some good ideas for how it could happen, banging transactions into what is essentially an in-memory analytic database. (I dispute SAP’s claims of transformational database technology leadership, but that doesn’t mean the underlying ideas aren’t good.)
• For those who can afford the associated technology disruption, memory-centric object-oriented DBMS could be appealing.
• Web scalability best practices commonly include keeping data in RAM (e.g., that’s pretty much the point of caching layer memcached).
• SaaS (Software as a Service) companies — such as Workday — often bring a particular tenant’s database entirely into RAM.
• QlikView highlights the benefits of doing business intelligence in RAM.
• SAS HPA makes the argument that even “big data analytics” should sometimes be done in RAM.
• I don’t have particularly favorable opinions at this time about marketing strategies or momentum at Oracle TimesTen, IBM solidDB, or VoltDB, but those examples at least serve to illustrate that memory-centric OLTP DBMS have existed for years.
• Actually, SAP has at least two good ideas, if you count Sybase as part of SAP.

And here’s the kicker: Intel told me last year that CPUs are headed to 46-bit address spaces around mid-decade. Indeed, they hired me to help figure out if that was enough.* That multiplies out to 64 terabytes of RAM on a single server, chip costs permitting. So most of what we now think of as operational databases — and many of the analytic ones too — will fit in-memory, even if they run very large businesses.

*And did so without putting the discussion under any kind of NDA.

Likely consequences of all this include:

• Legacy apps will (eventually) be consolidated and virtualized in-memory. Their underlying databases will grow so slowly that eventually the cost of putting them in RAM will be too low to worry about.
• Expensive storage systems will (continue to) be irrelevant to database processing. Databases that don’t fit in RAM will typically be big enough to require the attention of a lot of CPUs — and in those cases the DBMS software itself will handle all the storage tasks.
• Major OLTP DBMS vendors, such as Oracle, will need alternate in-memory code lines, because disk-centric architectures are sub-optimal in-memory. Well, that’s what they have those big R&D budgets for.
• SaaS vendors and web businesses may not rely on today’s major OLTP DBMS vendors. (I was going to say “won’t” rather than “may not” until I recalled the likely M&A endgame.) Traditional enterprises may blanch at migrating away from their legacy DBMS environments, but the trade-offs are different for technology companies using DBMS as subsystems.

Of course, the same trends that make data-storing chips cheaper will make data-generating chips cheaper too. So, just as there are huge amounts of machine-generated data that you’d never pay to store in RAM, the same will still be true 10 years from now; the data volumes involved will just be a lot bigger. And thus there will still be plenty of very large analytic databases using relatively cheap forms of storage, perhaps even disk.

But OLTP and other short-request processing are likely to wind up in-memory. And the same may be true for a considerable amount of analytics, especially but not only if the analytics have a low-latency requirement.

• IBM is already serious about supporting multiple database management systems. DB2 on open systems is IBM’s flagship DBMS. But DB2 on mainframes and at least one flavor of Informix seem to be getting maintained and enhanced fairly seriously as well. And IBM has further DBMS products as well (e.g., DB/2 on the AS/400). There’s little reason to think IBM would orphan MySQL or any other DBMS product.
• IBM is very open-source-friendly. For a company that grew up for decades on proprietary software — and still is a huge software products vendor — IBM is very serious about open source. If you doubt that, I have two words for you: “Linux” and “Eclipse”.
• MySQL might finally get its industrial-strength act together. IBM is good at database management and good at open source. MySQL becoming a no-apologies transactional DBMS would obviously put pressure on Ingres, PostgreSQL, and EnterpriseDB, although there surely would be lots of happy talk about the open source DBMS market being validated, lifting all the vendors and so on. Also, a better MySQL could be bad news for Microsoft SQL Server too.
• Sun has a lot of DBMS partnerships right now. Obviously, Sun owns MySQL, and has partnerships with MySQL storage engine vendors such as Infobright and Kickfire. Sun also has a substantial partnership with Greenplum, and a Barneyesque* one with ParAccel. And of course Sun has strong working relationships with major database vendors such as Oracle and Sybase. What’s more, on a case-by-case basis, Sun may cooperate in the field with yet other DBMS sellers. E.g., I’ve confirmed at least one instance of a Sun sales rep recommending a Kognitio DBMS.
• IBM partners with outside DBMS vendors too. You’d think IBM’s gazillion DBMS product lines would be enough. But nooooo. I frequently hear rumblings of IBM’s hardware or services operations working with other DBMS products as well. (This is, of course, actually to their credit.)
• Short-term, there probably would be little effect on partnerships. Greenplum runs on Sun’s Thumper/Thor line of boxes. DB2 doesn’t, and certainly isn’t optimized for same. In the short term, to sell Thors, Sun would presumably continue to sell Greenplum.
• Longer-term, there could be a DBMS rationalization. DB2, Informix, MySQL + storage engines, and big independent vendors such as Oracle and Sybase would surely always get attention. That’s a lot. There might not be room for much mind share for many database products and vendors beyond that list.

*A Barney partnership is one in which two or more vendors get on stage and do a song and dance about how much they love each other, with little substance beyond that.

Related links

• Larry Dignan thinks the IBM/Sun deal is sensible and ripe to happen.
• Dana Gardner thinks otherwise.
• Matt Asay seems to agree that IBM understands the open source business.
• Before IBM acquired it, solidDB was scheduled to provide a serious MySQL transaction processing engine.

… a real-time inventory control system which has the following requirements — basically it needs to provide high write-through-rate, and it needs little if any indexing functionality.

1) a central control system records/updates the inventory data (number/weight and etc.) at each room/rack — there exist thousands of racks/rooms

2) sensors (at different rack) also report/update the temperature to the central control system, at rate of approximately 1~2 updates/min — as there are thousands of sensors, the update throughput needs to be high considering the scalability requirement.

3) When some problems happen, we need to roll back the logs (to replay the events and diagnose the root cause).

His questions included:

Memory-centric DBMS or complex event/stream processing (CEP)? Given that the purpose is to record data, and that he wants to record all data rather than engage in immediate data reduction, true DBMS seems like the way to go.

What are the good in-memory DBMS alternatives anyway? He thought McObject’s eXtremeDB was dominant in the market, which surprised me (although McObject does seem to have made a push in Asia). I know long-time leaders TimesTen and solidDB have, since their acquisitions by Oracle and IBM respectively, pulled back from the standalone market. (Their new owners are more interested in front-end caching for Oracle and DB2 respectively.) But I didn’t think the pullback had been that — as it were — extreme.

And while my correspondent didn’t ask this, I’ll add — should he maybe just go with an ordinary DBMS anyway? A couple thousand updates per minute isn’t that forbidding. On the other hand, it might be hard to achieve with conventional DBMS on super-cheap hardware. And a memory-centric alternative that only logs to disk in near-real-time might be plenty good enough for any analytics they want to do.

He was quite frank about wanting to get experience with leading-edge technology, with an eye to deploying it in other use cases.  So it’s reasonable to be pretty general in this whole discussion. With that as background — well, I’ve already given some of my thoughts.  So what are yours?

Before Solid and solidDB were acquired by IBM, one of the things Solid was proudest of was some embedded apps in which solidDB ran for years in boxes without keyboards or monitors.

I still think it’s a pity that IBM isn’t using solidDB as broadly as the technology deserves. Even so, this is a nice endorsement of MySQL for reliable zero-DBA mid-range use.

• IBM’s acquisition of Solid seems to suggest a focus on DB2 caching rather than the embedded market
• McObject actually has built up something of a customer list, as per the boilerplate on any of its press releases.

And they do seem to have some nice features, including Patricia tries (like solidDB), R-trees (for geospatial), and some kind of hybrid disk-centric/memory-centric operation.

1. He calls out RAM consumption as a challenge for this kind of architecture.
2. He points out that it’s a big advantage to have data called and used in the same address space.

Being based in RAM is obviously a huge part of the H-Store scheme. But so is having transaction execution be close to the database.

IBM now has both ObjectGrid and a memory-centric DBMS (solidDB) that they’ve been using as a front end for DBMS. Integration of the two could be pretty interesting.

• solidDB is actually a very interesting hybrid disk/in-memory memory-centric database management system. However, the press release announcing the deal makes it sound as if solidDB is in-memory only.
• That strongly suggests that IBM is buying Solid mainly to compete with Oracle TimesTen. As of last June, solidDB was already IBM’s TimesTen answer via a partnership; this deal just solidifies that arrangement.
• This probably isn’t good news for Solid’s MySQL engine. That’s a pity, since solidDB technically has the potential to be the best MySQL engine around.
• Notwithstanding IBM’s presumed intentions, Solid’s main market success historically is as an embedded system in telecommunications equipment, network software, and similar systems.
• Last year I wrote a white paper on memory-centric data management, showcasing four products. IBM now has bought two of them, namely Solid’s and Applix’s (via Cognos).
• Comparisons to IBM’s embedded Java DBMS Cloudscape are pointless. That’s just a failed product vs. solidDB or Sybase SQL Anywhere, and IBM long ago cut its losses.

Here’s a link to a webcast I did with Solid back in June.

Edit: As per the invitations to the 11:00 am conference call, the following — straight from the press release — is evidently the official IBM one-paragraph description of the acquisition.

The acquisition of Solid supports IBM’s global Information on Demand strategy by adding real-time data access capabilities to the company’s portfolio of database and information management offerings. Solid’s software uses in-memory database technology to quickly retrieve data from a computer’s memory (or RAM). Using this software, businesses can access and store data at speeds up to ten times faster than traditional disk database systems. Solid’s architecture also enables applications to recover from system failure rapidly and to require little disk space and almost no hands-on administration.

Edit: IDG has quotes confirming the emphasis on TimesTen competition. IDG also reminds us that MySQL 6.0 will have its own storage engine Falcon. I guess that’s pretty much do or die for MySQL now.

More info and sign-up can be found here.