solidDB
Analysis of hybrid memory-centric DBMS solidDB and its developer Solid Technology. Related subjects include:
- IBM, which acquired Solid Technology and hence solidDB
- Memory-centric data management
MySQL is being used in an IBM Lotus appliance
Apparently, IBM is rolling out an appliance for small businesses. MySQL is under the covers. The appliance won’t have a keyboard or monitor, so there won’t be a lot of database administration going on.
Before Solid and solidDB were acquired by IBM, one of the things Solid was proudest of was some embedded apps in which solidDB ran for years in boxes without keyboards or monitors.
I still think it’s a pity that IBM isn’t using solidDB as broadly as the technology deserves. Even so, this is a nice endorsement of MySQL for reliable zero-DBA mid-range use.
| Categories: DBMS product categories, IBM and DB2, Mid-range, MySQL, solidDB | Leave a Comment |
McObject eXtremeDB — a solidDB alternative
McObject — vendor of memory-centric DBMS eXtremeDB — is a tiny, tiny company, without a development team of the size one would think needed to turn out one or more highly-reliable DBMS. So I haven’t spent a lot of time thinking about whether it’s a serious alternative to solidDB for embedded DBMS, e.g. in telecom equipment. However:
- IBM’s acquisition of Solid seems to suggest a focus on DB2 caching rather than the embedded market
- McObject actually has built up something of a customer list, as per the boilerplate on any of its press releases.
And they do seem to have some nice features, including Patricia tries (like solidDB), R-trees (for geospatial), and some kind of hybrid disk-centric/memory-centric operation.
| Categories: GIS and geospatial, In-memory DBMS, McObject, Memory-centric data management, solidDB | 5 Comments |
IBM discontinues the solidDB MySQL engine
Last year, I thought that solidDB could at least potentially be an outstanding MySQL engine. But as per news posted on SourceForge last week, that’s not going to happen. At least, it’s not going to happen via any development efforts from IBM.
| Categories: IBM and DB2, Mid-range, MySQL, Open source, solidDB | 4 Comments |
ObjectGrid versus H-Store
Billy Newport of IBM sees a lot of similarities between his app-server-based product ObjectGrid and H-Store. In both cases, constrained tree schemas are assumed, and OLTP performance goodness ensues. A couple of points I noted on a quick skim through his blog:
- He calls out RAM consumption as a challenge for this kind of architecture.
- He points out that it’s a big advantage to have data called and used in the same address space.
Being based in RAM is obviously a huge part of the H-Store scheme. But so is having transaction execution be close to the database.
IBM now has both ObjectGrid and a memory-centric DBMS (solidDB) that they’ve been using as a front end for DBMS. Integration of the two could be pretty interesting.
| Categories: Cache, H-Store, IBM and DB2, Memory-centric data management, OLTP, Theory and architecture, solidDB | Leave a Comment |
IBM acquires SolidDB to compete with Oracle TimesTen
IBM is acquiring Solid Information Technology, makers of solidDB. Some quick comments:
- solidDB is actually a very interesting hybrid disk/in-memory memory-centric database management system. However, the press release announcing the deal makes it sound as if solidDB is in-memory only.
- That strongly suggests that IBM is buying Solid mainly to compete with Oracle TimesTen. As of last June, solidDB was already IBM’s TimesTen answer via a partnership; this deal just solidifies that arrangement.
- This probably isn’t good news for Solid’s MySQL engine. That’s a pity, since solidDB technically has the potential to be the best MySQL engine around.
- Notwithstanding IBM’s presumed intentions, Solid’s main market success historically is as an embedded system in telecommunications equipment, network software, and similar systems.
- Last year I wrote a white paper on memory-centric data management, showcasing four products. IBM now has bought two of them, namely Solid’s and Applix’s (via Cognos).
- Comparisons to IBM’s embedded Java DBMS Cloudscape are pointless. That’s just a failed product vs. solidDB or Sybase SQL Anywhere, and IBM long ago cut its losses.
| Categories: Cache, Cognos, IBM and DB2, In-memory DBMS, Memory-centric data management, MySQL, OLTP, Oracle TimesTen, Sybase, solidDB | 3 Comments |
Webinar Wednesday June 27 at 2:00 pm ET
I’m sorry for the short notice, but — well, never mind what the distractions have been. This Wednesday, at 2:00 pm Eastern time, I’m doing a webinar on behalf of Solid. The core subject is memory-centric OLTP data management. I will of course also cover some DBMS and memory-centric generalities.
More info and sign-up can be found here.
| Categories: In-memory DBMS, Memory-centric data management, OLTP, solidDB | Leave a Comment |
Memory-centric vs. conventional DBMS — a Solid difference
I had the chance to talk at length with Solid Information Technology tech guru Antoni Wolski about their memory-centric DBMS technology architecture. The most urgent topic was what made in-memory database managers inherently faster than disk-based ones that happened to have all the data in cache. But we didn’t really separate that subject from the general topic of how they made their memory-centric technology run fast, from its introduction in 2002 through substantial upgrades in the most recent release.
There were 4 main subtopics to the call:
1. Indexing structures that are very different from those of disk-based DBMS.
2. Optimizations to those indexing structures.
3. Optimizations to logging and checkpointing.
4. Miscellaneous architectural issues.
Read more
| Categories: In-memory DBMS, Memory-centric data management, OLTP, solidDB | 3 Comments |
SolidDB caching for DB2
It’s just at the proof-of-concept stage, but Solid has a nice write-up about SolidDB being used as a front-end cache for DB2. Well, it’s a marketing document, so of course there’s a lot of pabulum too, but interspersed there’s some real meat as well. Highlights include 40X throughput improvement and 1 millisecond average response time (something that clearly can’t be achieved with disk-centric technology alone).
Analogies to Oracle/TimesTen are probably not coincidental; this is exactly the upside scenario for the TimesTen acquisition, as well as being TimesTen’s biggest growth area towards the end of its stint as an independent company.
| Categories: Cache, IBM and DB2, Memory-centric data management, OLTP, Oracle, Oracle TimesTen, solidDB | 1 Comment |
SolidDB and MySQL 5.0 – how industrial-strength in OLTP?
MySQL 4.0 is an OLTP joke. MySQL 5.0, however, shows a lot of progress in terms of real transactions, foreign keys, referential integrity, triggers, stored procedures and so on. In anticipation of the MySQL user conference next week, I got a quick briefing from Paola Lubet and Murat Demiroglu at Solid Information Technology, whose SolidDB is one of the two transactional storage engines for MySQL (the other is InnoDB, now owned by Oracle).
The layer provided by MySQL actually does most of what I think of as “language processing” – parsing, optimization, drivers, triggers, stored procedures, referential integrity, etc. SolidDB is a storage engine providing actual execution. Its features and virtues include:
• Online backup. (Note: Apparently, the extra-cost InnoDB online backup product isn’t showing up on price lists these days.)
• Optimistic (as well as pessimistic) concurrency control. This can be a good performance feature for applications that have a whole lot of Adds and very few Changes.
• General reliability. Unless they really botched the port, Solid benefits from a long history of very reliable operation.
• High availability. Scheduled for alpha in early summer and beta in the fall is a high-availability option. This initial-release will be master-slave synchronous replication. More sophisticated replication could come later on, as could memory-centric performance, if market conditions seem to warrant it (I’m betting they will).
| Categories: Mid-range, MySQL, OLTP, Open source, solidDB | 2 Comments |
Naming the DBMS disruptors
Edit: This post has largely been superseded by this more recent one defining mid-range relational DBMS.
I find myself defining a new product category – midrange OLTP/multipurpose DBMS. (Or just midrange DBMS for brevity.) Nothing earthshaking here; I’m simply referring to those products that: Read more
Oracle, Tangosol, objects, caching, and disruption
Oracle made a slick move in picking up Tangosol, a leader in object/data caching for all sorts of major OLTP apps. They do financial trading, telecom operations, big web sites (Fedex, Geico), and other good stuff. This is a reminder that the list of important memory-centric data handling technologies is getting fairly long, including:
- Object caching (e.g., Tangosol, Progress ObjectStore)
- In-memory RDBMS (e.g., Oracle TimesTen, Solid BoostEngine, McObject eXtremeDB)
- Stream processing (e.g., Progress Apama, Streambase)
And that’s just for OLTP; there’s a whole other set of memory-centric technologies for analytics as well.
When one connects the dots, I think three major points jump out:
- There’s a lot more to high-end OLTP than relational database management.
- Oracle is determined to be the leader in as many of those areas as possible.
- This all fits the market disruption narrative.
I write about Point #1 all the time. So this time around let me expand a little more on #2 and #3.
OLTP database management system market – the consensus isn’t ALL wrong (deck-clearing post #1)
Most of what I’ve written lately about database management seems to have been focused on analytic technologies. But I have a lot to say on the OLTP (OnLine Transaction Processing) side too. So let’s start by clearing the decks. Here’s a list of some consensus views that I in essence agree with:
- Oracle is the top of the line, and has nothing wrong with it other than cost of ownership and the non-joys of doing business with Oracle Corporation.
- DB2/mainframe is a fine product, but only if you like IBM mainframes.
- DB2/open systems is another fine product, but it’s hard to think of reasons to use it over Oracle.
- Microsoft SQL Server has great cost of ownership if you’re a Windows (server) shop anyway, especially on the administrative side. It does most but not all of what Oracle does.
- Sybase Adaptive Server Enterprise is a lot like SQL Server, but without the Windows dependence or the great Microsoft tools. If you have it installed or are Chinese, you should strongly consider using it, but otherwise there are better alternatives.
- Progress’ DBMS is great if you don’t need any of the features it’s missing. Administration, for example, is a super-low-cost breeze. But why use it unless you’re also using the Progress development tools?
- Intersystems’ Cache’ is another fine mid-range product that involves buying into the vendors’ whole tool set – all the more so because it isn’t relational.
- Small-footprint embedded DBMS, from vendors such as Sybase’s iAnywhere division or Solid Information Technologies, are off in their own little world. Mainly, that world is telecom, with a satellite in medical devices, although other kinds of networked equipment also sometimes use these products.
- IBM’s non-DB2 database management products – IMS, Informix, etc. – are fine things to stick with until you have to change. Ditto products from Software AG, Computer Associates, Cincom, etc.
- MySQL Version 4 is an OLTP joke, but it’s a joke many people share. (Hey — a lot of blogs, including mine, run on Wordpress and MySQL 4.)
- Until Ingres is meaningfully marketed and sold outside its installed base, it’s not worth worrying about.
- PostgreSQL is more significant as the underpinning of other products — mainly EnterpriseDB in the OLTP space — than it is in its own right.
Memory-centric data management whitepaper
I have finally finished and uploaded the long-awaited white paper on memory-centric data management.
This is the project for which I origially coined the term “memory-centric data management,” after realizing that the prevalent “in-memory DBMS” creates all sorts of confusion about how and whether data persists on disk. The white paper clarifies and updates points I have been making about memory-centric data management since last summer. Sponsors included:
- Applix, vendors of in-memory/memory-centric MOLAP tool TM1
- Progress Software, vendors of ObjectStore, an OODBMS that has more impressive references in-memory or otherwise memory-centric than it does in classical disk-based configurations, and also of the Apama stream processing products
- SAP, vendors of the BI Accelerator functionality of SAP NetWeaver, or whatever tortured name they want to give it this month — basically, that’s a very cool in-memory columnar data mart technology
- Solid Information Technology, vendor of hybrid in-memory/disk-based OLTP RDBMS. Historically focused on the embedded systems market, especially telecom and networking, they’ve recently been in the news because of a deal with MySQL that is designed to extend their reach.
- Intel, makers of the processors used to run a lot of the other sponsors’ products (including all BI Accelerator installations to date).
If there’s one area in my research I’m not 100% satisfied with, it may be the question of where the true hardware bottlenecks to memory-centric data management lie (it’s obvious that the bottleneck to disk-centric data management is random disk access). Is it processor interconnect (around 1 GB/sec)? Is it processor-to-cache connections (around 5 GB/sec)? My prior pronouncements, the main body of the white paper, and the Intel Q&A appendix to the white paper may actually have slightly different spins on these points.
And by the way — the current hard limit on RAM/board isn’t 2^64 bytes, but a “mere” 2^40. But don’t worry; it will be up to 2^48 long before anybody actually puts 256 gigabytes under the control of a single processor.
| Categories: Cognos, Companies and products, In-memory DBMS, Intel, MOLAP, Memory-centric data management, Open source, Progress, Apama, and DataDirect, SAP AG, solidDB | 1 Comment |
Solid/MySQL fit and positioning
I felt like writing a lot about the great potential fit between MySQL and Solid over the weekend, but Solid didn’t want me to do so. Now, however, I’m not in the mood, so I’ll just say that in OLTP, Solid’s technology is strong where MySQL’s is weak, and vice-versa. E.g., Solid is so proud of its zero-administration capabilities that, without MySQL, it doesn’t have much in the way of admin tools at all. Conversely, I think that many of those websites that crash all the time with MySQL errors would crash less with the Solid engine underneath. (Solid happens to be proud of its BLOB-handling capability, efficiency-wise.)
Neither outfit is good in data warehousing, or in text search, image search, etc. (Solid slings big files around, but it doesn’t peer closely inside them). But for OLTP of tabular or dumb media data, this looks like a great fit.
Whether anybody will care, however, is a different matter.
Lisa Vaas of eWeek offers a survey of the many MySQL engine options.
EDIT: Another Lisa Vaas article makes it clear that MySQL is planning to compete in data warehousing/OLAP as well.
| Categories: Memory-centric data management, Mid-range, MySQL, OLTP, Open source, solidDB | 3 Comments |
More on Solid and MySQL?
In a stunningly self-defeating move, my friends at Solid have decided that anything about their already-leaked possible cooperation with MySQL is embargoed.
Indeed, they’ve emphasized to me multiple times that they do not wish me to write about it.
I shall honor their wishes. I hope they are pleased with the sophistication and insight of the coverage they receive from other sources.
| Categories: Memory-centric data management, Mid-range, MySQL, OLTP, Open source, solidDB | 3 Comments |
MySQL gets the Solid engine
Solid and MySQL have struck a deal (and for some odd reason I had to find out about it from Slashdot and then here rather than from one one the companies). Apparently Solid will open source a version of its storage engine, to be used with the MySQL front-end.
Solid’s core technology is a lightweight, zero-administration OLTP RDBMS. And they really mean “zero-administration,” because as they like to point out, a typical deployment is embedded in a piece of telecom equipment that doesn’t even have a keyboard. Now, that doesn’t really mean the Solid engine would still be zero-administration in other applications, but sure aren’t talking about something as prickly as, say, Oracle.
That said, Solid’s technology has its limitations. It isn’t historically designed for the query load (volume or mix) of, say, an SAP installation. It certainly doesn’t have much in the way of data warehousing functionality. And it doesn’t have much in the way of administration tools itself (although presumably MySQL will fill that gap).
One very important aspect of the Solid technology is its hybrid memory-centric design. Much more on that soon. My white paper on memory-centric data management is finally close to publication, with Solid as a co-sponsor. At some point I’ll even do a webinar for them associated with the paper.
I don’t know whether that’s part of the MySQL relationship — it would be very cool if it were.
| Categories: Memory-centric data management, Mid-range, MySQL, OLTP, Open source, solidDB | 2 Comments |
Defining and surveying “Memory-centric data management”
I’m writing more and more about memory-centric data management technology these days, including in my latest Computerworld column. You may be wondering what that term refers to. Well, I’ve basically renamed what are commonly called “in-memory DBMS,” for what I think is a very good reason: Most of the products in the category aren’t true DBMS, aren’t wholly in-memory, or both! Indeed, if you catch me in a grouchy mood I might argue that “in-memory DBMS” is actually a contradiction in terms.
I’ll give a quick summary of the vendors and products I am focusing on in this newly-named category, and it should be clearer what I mean:
- TimesTen (now owned by Oracle): TimesTen is the quintessentional “in-memory DBMS.” It’s a fairly full relational DBMS, but if you want to persist memory to disk it has to be handed off to a conventional DBMS. Historically, that has usually been MySQL or Oracle. TimesTen’s biggest market penetration has been in financial trading.
- Solid Information Technology’s BoostEngine: Solid is a Finnish company (or was — it’s pretty American now) specializing in embedded DBMS sold mainly for telecommunication uses. Big OEM customers include several well-known telecom equipment manufacturers and HP (for OpenView). “Embedded” often means no DBA, no monitor, no keyboard — they box manufacturer installs it and there it stays for the life of the product. Solid has to offer strong replication capabilities, since its products are often used in highly distributed (e.g., multiblade, multibox) environments. So it’s taken the next step and exploited the replication by allowing customers to use some instances of the product disklessly.
- Event-stream products from Streambase and Progress: The canonical application for event-stream products is automating financial trading decisions based on the flow of market information. Mike Stonebraker, the brains behind Streambase, has recently popularized the idea; Progress bought Apama, who actually have been in the business longer. These applications require even more speed than the financial trading apps that TimesTen handles, and they discard most of the information they look at. In-memory is the only way to go.
- Progress’s ObjectStore: ObjectStore comes from the company Object Design, which merged into Excelon, which was acquired by Progress. It’s really a toolkit for building DBMS and similar systems, which is why it’s at various times been marketed as an OODBMS and an XML DBMS, without a lot of success either way. But there have been a few sterling apps built in ObjectStore even so, including a key part of the Amazon bookstore Despite this limited market success, a significant fraction of Progress’s best engineering talent has moved over to the Real-Time Division to focus on ObjectStore and other memory-centric products. The memory-centric aspect of ObjectStore is this: ObjectStore’s big virtue is that it gets objects from disk to memory and vice-versa very efficiently, then distributes and caches them around a network as needed. This was originally invented for client/server processing, but works fine in a multi-server thin client setup as well. And object processing, of course, relies on a whole lot of pointers. And pointer-chasing is pretty much the worst way to deal with the disk speed barrier, unless you do it in main memory.
- Applix’s TM1: Like many companies in the analytics area, Applix has had trouble deciding whether it sells applications, BI system software, or both. But in any case its core technology is TM1, a memory-centric MOLAP offering. Traditional MOLAP products reside on the horns of a nasty dilemma: They rely on precalculation to give good performance, but that causes ghastly database explosion. Applix gets out of this problem by doing no precalculation whatsoever, loading the data into main memory, and executing all queries on the fly.
- SAP’s BI Accelerator: SAP is building out an elaborate technology stack with NetWeaver, especially in the BI area. One important aspect is that the full data warehouse is logically broken (or copied) into a series of data marts called “InfoCubes.” BI Accelerator takes the logical next step, loading an entire InfoCube into main memory. Almost every query is executed via a full table scan, which would be insane on disk but makes perfect sense when the data is already in RAM.
So there you have it. There are a whole lot of technologies out there that manage data in RAM, in ways that would make little or no sense if disks were more intimately involved. Conventional DBMS also try to exploit RAM and limit disk access, via caching; but generally the data access methods they use in RAM are pretty similar to those they use when going out to disk. So memory-centric systems can have a major advantage.
