OLTP
Analysis of database management systems designed with a focus on OTLP (OnLine Transaction Processing) uses.
Another round of discussion on in-memory OLTP data management
Oracle Exadata was pre-teased as “Extreme performance.” Some incorrect speculation shortly before the announcement focused on the possibility of OLTP without disk, which clearly would speed things up a lot. I interpret that in part as being wishful thinking. 
The most compelling approach I’ve seen to that problem yet is H-Store, which however makes some radical architectural assumptions. One point I didn’t stress in my earlier posts, but which turned out to be a deal-breaker for one early tire-kicker, is that to use H-Store you have to be able to shoehorn each transaction into its own stored procedure. Depending on how intricate your logic is, that might make it hard to port an existing app to H-Store.
Even for new apps, it could get in the way of some things you might want to do, such as rule-based processing. And that could be a problem. A significant fraction of the highest-performance OLTP apps are customer-facing, and customer-facing apps are one of the biggest areas where rule-based processing comes into play.
| Categories: H-Store, In-memory DBMS, Memory-centric data management, OLTP | Leave a Comment |
Sun’s Rock chip is going to revolutionize OLTP? Yeah, right.
Ted Dziuba offers a profane and passionate screed to the effect that it would be really, really wonderful if Sun’s forthcoming Rock chip magically revolutionized OLTP. His idea — if I may dignify it with that term — seems to be that by solving some programming issues in multithreading, Sun will achieve orders of magnitude performance improvements in DBMS processing, with MySQL as the beneficiary.
Frankly, I don’t know what in the world Dziuba is talking about, and I strongly suspect that neither does he. Wikipedia wasn’t terribly enlightening, except to point out that some of the ideas originated with Tom Knight, which is encouraging. Ars Technica has a decent article about the Rock chip, but it’s hard to find support for Dziuba’s enthusiasm in their more sober discussion.
| Categories: MySQL, OLTP | 4 Comments |
Open source in-memory DBMS
I’ve gotten email about two different open source in-memory DBMS products/projects. I don’t know much about either, but in case you care, here are some pointers to more info.
First, the McObject guys — who also sell a relational in-memory product — have an object-oriented, apparently Java-centric product called Perst. They’ve sent over various press releases about same, the details of which didn’t make much of an impression on me. (Upon review, I see that one of the main improvements they cite in Perst 3.0 is that they added 38 pages of documentation.)
Second, I just got email about something called CSQL Cache. You can read more about CSQL Cache here, if you’re willing to navigate some fractured English. CSQL’s SourceForge page is here. My impression is that CSQL Cache is an in-memory DBMS focused on, you guessed it, caching. It definitely seems to talk SQL, but possibly its native data model is of some other kind (there are references both to “file-based” and “network”.)
| Categories: Cache, DBMS product categories, In-memory DBMS, McObject, Memory-centric data management, OLTP, Object, Open source | 5 Comments |
ScaleDB presents The Revenge of the Pointer
The MySQL user conference is upon us, and hence so are MySQL-related product announcements, including storage engines. One such is Kickfire. ScaleDB — smaller and earlier-stage — is another.
In a nutshell, ScaleDB’s proposition is:
-
Innovative approach to indexing relational DBMS, providing performance advantages.
-
Shared-everything scale-up that ScaleDB believes will leapfrog the MySQL engine competition already in Release 1. (In my opinion, this is the least plausible part of the ScaleDB story.)
-
State-of-the-art me-too facilities for locking, logging, replication/fail-over, etc., also already in Release 1.
Like many software companies with non-US roots, ScaleDB seems to have started with a single custom project, using a Patricia trie indexing system. Then they decided Patricia tries might be really useful for relational OLTP as well. The ScaleDB team now features four developers, plus half-time or so “Chief Architect” involvement from Vern Watts. Watts seems to pretty much have been Mr. IMS for the past four decades, and thus surely knows a whole lot about pointer-based database management systems; presumably, he’s responsible for the generic DBMS design features that are being added to the innovative indexing scheme. On ScaleDB’s advisory board is PeopleSoft veteran Rick Berquist, about whom I’ve had fond thoughts ever since he talked me into focusing on consulting as the core of my business.*
*More precisely, Rick pretty much tricked me into doing a day of consulting for $15K, then revealed that’s what he’d done, expressing the thought that he’d very much gotten his money’s worth. But I digress …
ScaleDB has no customers to date, but hopes to be in beta by the end of this year. Angels and a small VC firm have provided bridge loans; otherwise, ScaleDB has no outside investment. ScaleDB’s business model thoughts include:
| Categories: Data models and architecture, Mid-range, MySQL, OLTP, Open source, ScaleDB, Theory and architecture | Leave a Comment |
EnterpriseDB unveils Postgres Plus
EnterpriseDB is making a series of moves and announcements. Highlights include:
- Renaming/repositioning the product as “Postgres Plus.” The free product is now Postgres Plus, while the version you pay EnterpriseDB for is now Postgres Plus Advanced Server.
- Repackaging the products, so that Postgres Plus Advanced Server is a strict superset of Postgres Plus.
- New features added to Postgres Plus Advanced Server.
- Features newly migrated from Advanced Server down to Postgres Plus.
- A strategic investment by IBM.
- Stressing Postgres in EnterpriseDB marketing, and dropping the tag-line defining themselves as “the Oracle-compatible database company.”
So far as I can tell, most of the technical differences between Advanced Server and regular Postgres Plus lie in three areas: Read more
| Categories: Cache, Emulation, transparency, portability, EnterpriseDB and Postgres Plus, Mid-range, MySQL, OLTP, Open source, PostgreSQL | 1 Comment |
The core challenges of OLTP are changing
I wrote a few weeks ago about the H-Store project, which rejects a variety of assumptions underlying traditional OLTP database design. One of these is long transactions over open database connections. The idea is that the most demanding OLTP applications run on the Web, where abandonment is common, and hence the only sensible option is to break things up into simple chunks. Read more
| Categories: Application areas, OLTP | Leave a Comment |
More Twitter weirdness
Twitter commonly has the problem of duplicate tweets. That is, if you post a message, it shows up twice. After a little while, the dupe disappears, but if you delete the dupe manually, the original is gone too.
I presume what’s going on is that tweets are cached, the tweets are eventually batched to disk, and they don’t always get deleted from cache until some time after they’re persisted. If you happen to check the page of your recent tweets inbetween — boom, you get two hits. But what I don’t understand is why the two versions have different timestamps.
Presumably, this could be explained at a MySQL User Conference session next month, one of whose topics will be Intelligent caching strategies using a hybrid MemCache / MySQL approach. I’m so glad they don’t use stupid strategies to do this … Read more
| Categories: Cache, MySQL, OLTP, Specific users | 3 Comments |
eBay OLTP architecture
I’ve posted a couple times about eBay’s analytics side. As a companion, Don Burleson pointed me at a fascinating November, 2006 slide presentation outlining eBay’s transactional architecture and evolution. Highlights include:
- A whole lot of manual slicing of Oracle databases, so as not to exceed their capacity.
- A whole lot of careful design and ordering of transactions.
- Putting all the business logic in the application tier, with a custom O/R mapper. There’s lots of caching there, but very little state.
The presentation has a bunch of specific numbers, in case anybody wants to dive in.
| Categories: OLTP, Specific users, eBay | Leave a Comment |
ObjectGrid versus H-Store
Billy Newport of IBM sees a lot of similarities between his app-server-based product ObjectGrid and H-Store. In both cases, constrained tree schemas are assumed, and OLTP performance goodness ensues. A couple of points I noted on a quick skim through his blog:
- He calls out RAM consumption as a challenge for this kind of architecture.
- He points out that it’s a big advantage to have data called and used in the same address space.
Being based in RAM is obviously a huge part of the H-Store scheme. But so is having transaction execution be close to the database.
IBM now has both ObjectGrid and a memory-centric DBMS (solidDB) that they’ve been using as a front end for DBMS. Integration of the two could be pretty interesting.
| Categories: Cache, H-Store, IBM and DB2, Memory-centric data management, OLTP, Theory and architecture, solidDB | Leave a Comment |
The architectural assumptions of H-Store
I wrote yesterday about the H-Store project, the latest from the team of researchers who also brought us C-Store and its commercialization Vertica. H-Store is designed to drastically improve efficiency in OLTP database processing, in two ways. First, it puts everything in RAM. Second, it tries to gain an additional order of magnitude on in-memory performance versus today’s DBMS designs by, for example, taking a very different approach to ensuring ACID compliance.
Today I had the chance to talk with two more of the H-Store researchers, Sam Madden and Daniel Abadi.
| Categories: Database diversity, H-Store, In-memory DBMS, Memory-centric data management, OLTP | 3 Comments |
Mike Stonebraker calls for the complete destruction of the old DBMS order
Last week, Dan Weinreb tipped me off to something very cool: Mike Stonebraker and a group of MIT/Brown/Yale colleagues are calling for a complete rewrite of OLTP DBMS. And they have a plan for how to do it, called H-Store, as per a paper and an associated slide presentation.
| Categories: Database diversity, H-Store, In-memory DBMS, Memory-centric data management, Michael Stonebraker, OLTP, Theory and architecture | 30 Comments |
Mike Stonebraker’s DBMS taxonomy
In a response to my recent five-part series on DBMS diversity, Mike Stonebraker has proposed his own taxonomy of data management technologies over on Vertica’s Database Column blog.
- OLTP DBMSs focused on fast, reliable transaction processing
- Analytic/Data Warehouse DBMSs focused on efficient load and ad-hoc query performance
- Science DBMSs — after all MatLab does not scale to disk-sized arrays
- RDF stores focused on efficiently storing semi-structured data in this format
- XML stores focused on semi-structured data in this format
- Search engines — the big players all use proprietary engines in this area
- Stream Processing Engines focused on real-time StreamSQL
- “Lean and Mean,” less-than-a-database engines focused on doing a small number of things very well (embedded databases are probably in this category)
- MapReduce and Hadoop — after all Google has enough “throw weight” to define a category
He goes on to say that each will be architected differently, except that — as he already convinced me back in July — RDF will be well-managed by specialty data warehouse DBMS. Read more
| Categories: Data types, Database diversity, Michael Stonebraker, Mid-range, OLTP, RDF and graphs, Theory and architecture | 2 Comments |
Database management system choices — mid-range-relational
This is the fourth of a five-part series on database management system choices. For the first post in the series, please click here.
The other threat to the high-end relational DBMS vendors aims squarely at the heart of their business. It’s the mid-range relational database management systems, which are doing an ever-larger fraction of what their high-end cousins can. That said, different products do different things well. So if you’re not blindly paying up for the security of an all-things-to-all-people high-end DBMS, there are a number of factors you might want to consider.
| Categories: Database diversity, EnterpriseDB and Postgres Plus, Mid-range, MySQL, OLTP, PostgreSQL, Theory and architecture | 2 Comments |
Database management system choices – 4 categories of relational
This is the second of a five-part series on database management system choices. For the first post in the series, please click here.
For the most part, relational database management systems divide into four major classes:
- High-end OLTP (OnLine Transaction Processing) relational DBMS. Oracle is the flagship for this category, followed by DB2.
- Specialty data warehouse DBMS. Teradata is the leader here, followed by Netezza, DATAllegro, ParAccel, Vertica, Infobright, Greenplum, Kognitio, Sybase IQ, and a host of others.
- Mid-range relational database management systems. Most of the contenders here fall into one or more of three categories: Open-source-based relational DBMS (MySQL, PostgreSQL, EnterpriseDB); reseller-focused relational DBMS (Progress OpenEdge, Pervasive PSQL); or crippled “editions” of high-end systems. Microsoft SQL Server was once a clear mid-range system, but now is better classified as high-end OLTP.
- Embedded relational database management systems. The leader of this category is Sybase’s SQL Anywhere. Also significant are memory-centric products Oracle TimesTen and solidDB.
| Categories: Database diversity, OLTP, Theory and architecture | 9 Comments |
EnterpriseDB on Elastra, early stages
I finally caught up with Bob Zurek about EnterpriseDB’s foray into the Elastra cloud. Here are some highlights:
- There have been dozens of applicants for the EnterpriseDB/Elastra beta program. As is usual in limited beta programs, EnterpriseDB is trying to sort out the ones who’ll make a big commitment from the tire-kickers.
- The main interest in EnterpriseDB/Elastra has come from ISVs, and secondarily from purely online businesses (e.g., SaaS vendors, web businesses, and a large MMO game vendor). There’s been a little interest from enterprises.
- Significant fractions of the EnterpriseDB/Elastra beta applications come from each of the Oracle, PostgreSQL, and MySQL user communities. A few come from SQL Server. None come from DB2.
- Bob praised Elastra for its technology in clustering, starting/stopping instances, etc. He also said that EnterpriseDB had “educated” Elastra on EnterpriseDB internals and/or admin tools, to make the integration work.
- EnterpriseDB will start turning on a few beta Elastra customers any day now (i.e., it may well not take until March, the original target).
| Categories: Cloud computing, Elastra, EnterpriseDB and Postgres Plus, Mid-range, OLTP, Open source | Leave a Comment |
PostgreSQL speeds up OLTP
The Register reports on PostgreSQL 8.3, and emphasizes OLTP speedups and reductions in administrative burden:
Among the changes, Heap Only Tuples (HOT) that may cut the maintenance overhead of frequently updated tables by up to 75 per cent, spread checkpoints and background writer autotuning to reduce the impact of check points on response times, and an asynchronous commit option that also speeds the response times of certain transactions.
I wonder how EnterpriseDB compares on these features.
Edit: Slashdot has discussion and links. And here’s a PostgreSQL feature matrix.
| Categories: EnterpriseDB and Postgres Plus, Mid-range, OLTP, Open source, PostgreSQL | 1 Comment |
Why not database SaaS?
After a flurry of recent announcements of database SaaS (Software as a Service), eWeek has published a backlash article. The angle is that database SaaS is too expensive, because you can get decent DBMS for free and per-gig usage charges might be expensive for big databases.
I think that’s missing the point. Most OLTP databases are pretty small. Or, if they’re big, they get that way through a lot of transactions. In the first case, hosted management is cheap. In the second case, hosted management is taking care of a large burden for you. Read more
| Categories: Kognitio, OLTP, Software as a Service (SaaS) | 2 Comments |
EnterpriseDB joins Elastra in the Amazon cloud
When Elastra announced their service to host MySQL and PostgreSQL in the Amazon S3/EC2 cloud, I immediately told my dear darling clients at EnterpriseDB they should do the same. Whereupon they told me it would happen soon. However, they neglected to tell me when it was actually announced. So I know no more than can be found in this Computerworld article.
But I’ll say this — it’s a very tempting option, both for new web-based applications or businesses, or simply as a development platform pending later redeployment.
| Categories: Amazon and its cloud, Cloud computing, Elastra, EnterpriseDB and Postgres Plus, Mid-range, OLTP, Open source, Software as a Service (SaaS) | 2 Comments |
What hard-core transactional applications have actually been built in MySQL, PostgreSQL, EnterpriseDB, or FileMaker?
And here’s the biggie.
Question of the day #3
What complex, high-volume transactional applications have actually been built in mid-range DBMS such as MySQL, PostgreSQL, FileMaker, or EnterpriseDB?
I’ve been flamed for suggesting that MySQL or FileMaker aren’t fully equal to Oracle and DB2 in supporting hard-core transactional applications. (Which is ironic, because I’ve also been flamed for suggesting hard-core transactional support isn’t as big a deal for DBMS selection as some relational purists insist. But I digress …) So I’m putting the question out there — what impressive transactional applications do the stand-alone mid-range DBMS actually support? Read more
| Categories: EnterpriseDB and Postgres Plus, FileMaker, Mid-range, MySQL, OLTP, Open source, PostgreSQL | 20 Comments |
14 reasons not to use MySQL or other mid-range database management systems
I may argue for the use of open source and other mid-range database management systems, but a lot of industry sentiment remains on the other side. Vendors of high-end RDBMS naturally advocate enterprise-wide single-vendor adoption. Many CIOs and industry analysts, overwhelmed by product proliferation, think that’s a neat idea as well.
And in fairness, they’re not entirely wrong. Here are 14 reasons for using high-end relational database management systems, even on applications for which mid-range DBMS would suffice. Read more
| Categories: Microsoft and SQL*Server, Mid-range, MySQL, OLTP, Open source, Oracle, PostgreSQL | 19 Comments |
The world according to Derek Rodner of EnterpriseDB
If you’re interested in the world of mid-range, OLTP, and/or open source database management systems, Derek Rodner’s blog is worth checking out. His 2007 Year in Review post deserves a look — even though it’s about as unbiased and spin-free as Bill O’Reilly’s TV show, in that combines multiple shots each at Oracle and MySQL with some plugs for EnterpriseDB. I’ve already praised his post a month ago listing large numbers of EnterpriseDB successes. Of course there are multiple heartfelt arguments on behalf of Postgres (too many to link to specifically). And he even has a great set of tips — which I hereby recommend to all my vendor clients — on how best to use Google AdWords.
| Categories: EnterpriseDB and Postgres Plus, Mid-range, OLTP, Open source, PostgreSQL | 1 Comment |
Optimizer geekdom
I’ve been a DBMS analyst since before there were cost-based optimizers or, for that matter, a whole lot of relational DBMS. And in the 20 years that optimizers have been around, I’ve never fully understood why they’re so simple-minded. Even so, I think they’re pretty cool, as per the fanboyish discussion in this 2004 Computerworld column.
So I’m delighted to see that the Oracle folks have started a hardcore blog on optimizer details. If you want to get a sense of how smart a leading DBMS is or isn’t, I encourage you to check it out.
Keep getting great research about database management and related technologies. Sign up today! No hassle, no spam!
IBM acquires SolidDB to compete with Oracle TimesTen
IBM is acquiring Solid Information Technology, makers of solidDB. Some quick comments:
- solidDB is actually a very interesting hybrid disk/in-memory memory-centric database management system. However, the press release announcing the deal makes it sound as if solidDB is in-memory only.
- That strongly suggests that IBM is buying Solid mainly to compete with Oracle TimesTen. As of last June, solidDB was already IBM’s TimesTen answer via a partnership; this deal just solidifies that arrangement.
- This probably isn’t good news for Solid’s MySQL engine. That’s a pity, since solidDB technically has the potential to be the best MySQL engine around.
- Notwithstanding IBM’s presumed intentions, Solid’s main market success historically is as an embedded system in telecommunications equipment, network software, and similar systems.
- Last year I wrote a white paper on memory-centric data management, showcasing four products. IBM now has bought two of them, namely Solid’s and Applix’s (via Cognos).
- Comparisons to IBM’s embedded Java DBMS Cloudscape are pointless. That’s just a failed product vs. solidDB or Sybase SQL Anywhere, and IBM long ago cut its losses.
| Categories: Cache, Cognos, IBM and DB2, In-memory DBMS, Memory-centric data management, MySQL, OLTP, Oracle TimesTen, Sybase, solidDB | 3 Comments |
Elastra - somewhat more sensible Amazon-based DBMS option
Elastra is a startup offering MySQL and PostgreSQL SaaS instances in the Amazon S3/EC2 cloud. On their board is John Hummer, which I generally regard as a good thing, although it’s hardly a guarantee of success.* High Scalability raises some doubts about Elastra’s pricing, but I think that may be missing the point. Read more
| Categories: Amazon and its cloud, Cloud computing, Elastra, MySQL, OLTP, Open source, PostgreSQL, Software as a Service (SaaS) | 2 Comments |
Amazon SimpleDB - when less is, supposedly, enough
I’ve posted several times about Amazon as an innovative, super-high-end user — doing transactional object caching with ObjectStore, building an inhouse less-than-DBMS called Dynamo, or just generally adopting a very DBMS2-like approach to data management. Now Amazon is bring the Dynamo idea to the public, via a SaaS offering called SimpleDB. (Hat tip to Tim Anderson.)
SimpleDB is obviously meant to be a data server for online applications. There are no joins, and queries don’t run over 5 seconds, so serious analytics are out of the question. Domains are limited to 10GB for now, so extreme media file serving also isn’t what’s intended; indeed, Amazon encourages one to use SimpleDB to store pointers to larger objects stored as files in Amazon S3.
On the other hand, if you think of SimpleDB as an OLTP DBMS, your head might explode. There’s no sense of transaction, no mechanisms to help with integrity, no way to do arithmetic, and indeed no assurance that writes will be immediately reflected in reads. Read more
