What hard-core transactional applications have actually been built in MySQL, PostgreSQL, EnterpriseDB, or FileMaker?
And here’s the biggie.
Question of the day #3
What complex, high-volume transactional applications have actually been built in mid-range DBMS such as MySQL, PostgreSQL, FileMaker, or EnterpriseDB?
I’ve been flamed for suggesting that MySQL or FileMaker aren’t fully equal to Oracle and DB2 in supporting hard-core transactional applications. (Which is ironic, because I’ve also been flamed for suggesting hard-core transactional support isn’t as big a deal for DBMS selection as some relational purists insist. But I digress …) So I’m putting the question out there — what impressive transactional applications do the stand-alone mid-range DBMS actually support?
We can’t ask that question of the crippled editions of Oracle, DB2, et al.; it would be too hard to sort out from the apps running on their high-end versions. We needn’t ask the question of Progress and Intersystems; their reseller catalogs give the answer. We certainly needn’t ask it of MaxDB; it runs a significant fraction of the SAP user base. But for the four DBMS I named, I think it’s an interesting challenge.
Specific information I hope you’ll provide includes:
- Peak load of concurrent users.
- Peak load of transactions/second and/or updates/second.
- Metrics for the complexity of the database (e.g., number of tables).
- How intensely the application uses triggers, stored procedures, and/or declarative referential integrity.
- Any other metrics you think are impressive or relevant.
My logistical suggestions for supplying examples are as per Question of the day #1.
Comments
20 Responses to “What hard-core transactional applications have actually been built in MySQL, PostgreSQL, EnterpriseDB, or FileMaker?”
Leave a Reply
When you talk of MySQL, do you also talk of Brighthouse?
I don’t know whether YouTube is “transactional” in the classical sense of the term… but it is built on top of MySQL for sure and it does serve quite a volume of transactions. However, they use memcached on top of it, I believe.
I believe that slashdot.org is build upon MySQL using InnoDB for transactions. I don’t know the current size or any specifics, but it does have good response times and appears to handle heavy loads very well.
http://mysqldatabaseadministration.blogspot.com/2008/01/i-will-not-blob.html
Hi Casey, Slashdot uses memcached as well, I think.
But more exciting than Slashdot is Wikipedia which is the 2005 user of the year (http://www.mysql.com/why-mysql/awards/app-of-the-year.html). Again, wikipedia relies on memcached too! Here is a quote:
“Wikipedia is the most popular wiki in the world, operating in more than 50 languages. Wikipedia uses MySQL to process more than 200 million queries and 1.2 million updates per day.”
Well. With Wikipedia, we may very well have a winner. How can top that as far as high profile MySQL users go?
Although I am not able to give away the name of the company, I asked for some numbers from their Senior Linux Systems Administrator today to show some volume numbers for you, Curt. The company is a “large SaaS company which processes data in near real-time”:
==quote==
We have over 1PB of MySQL data on-line. Over 10,000 servers (Linux,
red-hat based, 386 based systems from a major vendor). Only about 7,000
of the servers are running MySQL. We have a ‘small team’ of system
administrators that manage the entire system. I wish I could give you
the exact number publicly. Let’s just say the server-to-admin ratio is
jaw-dropping.
Customer data comes in via a web farm, and is quickly passed to an
application processing layer (MySQL). Results are then stored in
short-term high-performance storage (MySQL). And finally the data is
moved to long-term data-warehouse storage (MySQL). The system is capable
of handling millions of requests every second.
We use a little bit of InnoDB, but due to the storage costs we don’t use
it wholesale. We use ‘specialized techniques’ to avoid table lock
contention issues common with MyISAM. (Sorry, can’t give that one away.)
More than 8 billion front-end transactions daily, and more than 320
billion MySQL queries daily.
==end quote==
Not sure if this is what you are looking for, Curt. In fact, not sure if there are many truly homogeneous “hard-core transactional applications” out there any more. But, you asked for “complex, high-volume transactional applications”, and I challenge you to find systems doing 320,000,000,000 queries per day. That’s pretty high volume, IMHO…
Cheers,
Jay
p.s. And, I know you were looking for an apology (re: your comment on Roland’s blog). I apologize for anything untrue I may have said. Perhaps you may write a blog post detailing how you receive revenue from your clients and in what ways your client’s marketing requests are different from your blog.
Jay,
http://www.monash.com is my business site; you can see my business model there. In particular, http://www.monash.com/writing.html describes ALL the sponsored writing I do. Either it’s billed as a white paper, or it’s not sponsored.
I refer you in particular to the last bullet point, where I stress that the papers are “truthfully marketed as independent.”
The other area of overlap is when I say to somebody “As an analyst and blogger I’d like you to answer this tough question, and as a consultant I advise you to give in, for your own sake.” If I publicly castigate a client for their secrecy, then later praise them for opening up, you can assume that I repeated in private what I also was saying in public.
Best,
CAM
Re Slashdot, Wikipedia, and the others referred to in this thread so far — yep, those are decidedly non-trivial. But I’m guessing that the database schemas are actually pretty simple, no? And there isn’t much use of the specific kinds of features I called out in the initial post, is there?
As I already mentioned above: A couple years ago — in a series of articles that gave the name to this blog — I inspired flames from the other side by pointing out those features weren’t as relevant to new applications today as they once were.
CAM
Re Infobright: I recently posted several things about them, which may be found via http://www.dbms2.com/category/products-and-vendors/infobright-brighthouse/ But in this thread I’m asking about transactional apps.
CAM
Curt, first, why don’t you tattoo infobright? it will save you some keystrokes from having to plug them everywhere.
“But I’m guessing that the database schemas are actually pretty simple, no? ”
You think that no complex site can run with simple database schemas? That’d be gross ignorance.
Frank
Frank,
I can’t think of a better way of disagreeing with you than to simply repeat what I’ve already said in this thread.
So I’ll save myself the typing and let it go.
CAM
Curt,
OK, thanks for pointing to your services and explaining things; that is appreciated.
Re: whether the company I mentioned above uses triggers, SPs, and declarative integrity, I’m not sure at all. I can inquire, though.
Cheers,
Jay
p.s. It might assist others looking to answer your post question if you gave us an example of something you’re thinking about. Are you talking about financials? ERPs? Something else?
Jay,
There are two intertwined sets of issues here:
1. Does a DBMS have what it takes to support traditional, complex OLTP applications?
2. Does it matter whether a DBMS has what it takes to support traditional, complex OLTP applications?
If I had to pick, I’d say the second one is more interesting, even though in this thread I’m focusing somewhat more on the first.
Obviously, an extreme “No” to #2 is ridiculous, but even so opinions can differ a lot. As they used to say at the Kennedy School, where you stand depends upon where you sit.
As for what would be a complex OLTP app — well, ERP and airline reservations are probably the top two examples. The latter is what Dan Weinreb posted about in another thread. The former is why I say that anything with SAP certification — something MySQL so far has been unable to achieve — is ipso facto capable of handling them. But I’m trying not to be that simplistic.
Besides, SAP itself doesn’t really believe in complex relational data modeling any more for its new application development. I’ve posted about that several times, I think.
CAM
I don’t have the detailed first-hand knowledge you’re looking for, but for some second hand info check out ’12 Days of Scale out’
http://www.mysql.com/news-and-events/?year=2007
And less recently, Sabre is the highest profile non-wikipedia,-yahoo,or-google mysql user that I know of.
http://www.mysql.com/news-and-events/press-release/release_2003_33.html
You should be able to find case studies w/stats for Sabre
Holdings online, because I believe their conversion from Oracle was celebrated at more than one mysql/oss user conferences. Note that they used Golden Gate for replication when they decloaked, but I still count that as a huge MySQL success.
Facebook and Youtube (and yahoo/google) are mammoth mysql deployments but I believe sizing details have not been published. I have seen/heard several conference video/podcasts on these but they’ve been cagey on the details.
Oh yeah — dotlrn.org is an LMS that runs on postgresql and is heavy-duty transactional using triggers/DRI/etc.
I know they have multiple 10K+ user deployments, though you’d could get better stats by reaching out on their bboards.
John,
That dotlrn.org one looks interesting. Thanks!
I’ll add them to the tags on the original post and see if anybody magically shows up on reindexing. 😉
Interesting set of statistics from various comments.
The 8 billion transactions per day is some kind of telco or financial company. If it was a brokerage given the data, the 7000 or so mysql instances would equate to one instance per equity/stock.
At least seeing what other vendors with large web traffic have done, it would suggest you would shard on each equity. If it was a telco, shard on each district/street code. Then aggregrate into another database.
In terms of the usage outlined by Jay none of our mysql customers come close, though I will check where they do rank. The day job I am talking about…
Have Fun
Paul
Paul,
Your comment on brokerage reminds me of the fantasy stock market http://www.blogshares.com Data corruption is so common they have a UI feature that adds up total stockholdings, compares to shares outstanding, and notes the difference between the two figures that of course should ALWAYS be identical.
If ACID is what one is looking for, Blogshares has pH around 12. Over a serious DBMS, it would take real effort to build an app that bad.
I greatly doubt the Blogshares folks used InnoDB.
CAM
[…] is perhaps what Curt Monash of DBMS2 does when he groups it (and others) with FileMaker. He asks, “What hard-core transactional applications have actually been built in MySQL, PostgreSQL, Ente…. And he gets plenty of […]
[…] paio di letture da consigliare questa settimana, la prima del solito DBMS2 dal titolo ” What hard-core transactional applications have actually been built in MySQL, […]