January 28, 2008

What hard-core transactional applications have actually been built in MySQL, PostgreSQL, EnterpriseDB, or FileMaker?

And here’s the biggie.

Question of the day #3

What complex, high-volume transactional applications have actually been built in mid-range DBMS such as MySQL, PostgreSQL, FileMaker, or EnterpriseDB?

I’ve been flamed for suggesting that MySQL or FileMaker aren’t fully equal to Oracle and DB2 in supporting hard-core transactional applications. (Which is ironic, because I’ve also been flamed for suggesting hard-core transactional support isn’t as big a deal for DBMS selection as some relational purists insist. But I digress …) So I’m putting the question out there — what impressive transactional applications do the stand-alone mid-range DBMS actually support?

We can’t ask that question of the crippled editions of Oracle, DB2, et al.; it would be too hard to sort out from the apps running on their high-end versions. We needn’t ask the question of Progress and Intersystems; their reseller catalogs give the answer. We certainly needn’t ask it of MaxDB; it runs a significant fraction of the SAP user base. But for the four DBMS I named, I think it’s an interesting challenge.

Specific information I hope you’ll provide includes:

My logistical suggestions for supplying examples are as per Question of the day #1.

Comments

20 Responses to “What hard-core transactional applications have actually been built in MySQL, PostgreSQL, EnterpriseDB, or FileMaker?”

  1. Adrien on January 28th, 2008 9:26 am

    When you talk of MySQL, do you also talk of Brighthouse?

  2. Daniel Lemire on January 28th, 2008 10:28 am

    I don’t know whether YouTube is “transactional” in the classical sense of the term… but it is built on top of MySQL for sure and it does serve quite a volume of transactions. However, they use memcached on top of it, I believe.

  3. Casey on January 28th, 2008 11:10 am

    I believe that slashdot.org is build upon MySQL using InnoDB for transactions. I don’t know the current size or any specifics, but it does have good response times and appears to handle heavy loads very well.

  4. Frank Mash on January 28th, 2008 11:32 am
  5. Daniel Lemire on January 28th, 2008 2:47 pm

    Hi Casey, Slashdot uses memcached as well, I think.

    But more exciting than Slashdot is Wikipedia which is the 2005 user of the year (http://www.mysql.com/why-mysql/awards/app-of-the-year.html). Again, wikipedia relies on memcached too! Here is a quote:

    “Wikipedia is the most popular wiki in the world, operating in more than 50 languages. Wikipedia uses MySQL to process more than 200 million queries and 1.2 million updates per day.”

    Well. With Wikipedia, we may very well have a winner. How can top that as far as high profile MySQL users go?

  6. Jay Pipes on January 28th, 2008 3:30 pm

    Although I am not able to give away the name of the company, I asked for some numbers from their Senior Linux Systems Administrator today to show some volume numbers for you, Curt. The company is a “large SaaS company which processes data in near real-time”:

    ==quote==

    We have over 1PB of MySQL data on-line. Over 10,000 servers (Linux,
    red-hat based, 386 based systems from a major vendor). Only about 7,000
    of the servers are running MySQL. We have a ‘small team’ of system
    administrators that manage the entire system. I wish I could give you
    the exact number publicly. Let’s just say the server-to-admin ratio is
    jaw-dropping.

    Customer data comes in via a web farm, and is quickly passed to an
    application processing layer (MySQL). Results are then stored in
    short-term high-performance storage (MySQL). And finally the data is
    moved to long-term data-warehouse storage (MySQL). The system is capable
    of handling millions of requests every second.

    We use a little bit of InnoDB, but due to the storage costs we don’t use
    it wholesale. We use ‘specialized techniques’ to avoid table lock
    contention issues common with MyISAM. (Sorry, can’t give that one away.)

    More than 8 billion front-end transactions daily, and more than 320
    billion MySQL queries daily.

    ==end quote==

    Not sure if this is what you are looking for, Curt. In fact, not sure if there are many truly homogeneous “hard-core transactional applications” out there any more. But, you asked for “complex, high-volume transactional applications”, and I challenge you to find systems doing 320,000,000,000 queries per day. That’s pretty high volume, IMHO…

    Cheers,

    Jay

    p.s. And, I know you were looking for an apology (re: your comment on Roland’s blog). I apologize for anything untrue I may have said. Perhaps you may write a blog post detailing how you receive revenue from your clients and in what ways your client’s marketing requests are different from your blog.

  7. Curt Monash on January 28th, 2008 4:41 pm

    Jay,

    http://www.monash.com is my business site; you can see my business model there. In particular, http://www.monash.com/writing.html describes ALL the sponsored writing I do. Either it’s billed as a white paper, or it’s not sponsored.

    I refer you in particular to the last bullet point, where I stress that the papers are “truthfully marketed as independent.”

    The other area of overlap is when I say to somebody “As an analyst and blogger I’d like you to answer this tough question, and as a consultant I advise you to give in, for your own sake.” If I publicly castigate a client for their secrecy, then later praise them for opening up, you can assume that I repeated in private what I also was saying in public.

    Best,

    CAM

  8. Curt Monash on January 28th, 2008 4:46 pm

    Re Slashdot, Wikipedia, and the others referred to in this thread so far — yep, those are decidedly non-trivial. But I’m guessing that the database schemas are actually pretty simple, no? And there isn’t much use of the specific kinds of features I called out in the initial post, is there?

    As I already mentioned above: A couple years ago — in a series of articles that gave the name to this blog — I inspired flames from the other side by pointing out those features weren’t as relevant to new applications today as they once were.

    CAM

  9. Curt Monash on January 28th, 2008 4:47 pm

    Re Infobright: I recently posted several things about them, which may be found via http://www.dbms2.com/category/products-and-vendors/infobright-brighthouse/ But in this thread I’m asking about transactional apps.

    CAM

  10. Frank Mash on January 28th, 2008 5:57 pm

    Curt, first, why don’t you tattoo infobright? it will save you some keystrokes from having to plug them everywhere.

    “But I’m guessing that the database schemas are actually pretty simple, no? ”

    You think that no complex site can run with simple database schemas? That’d be gross ignorance.

    Frank

  11. Curt Monash on January 28th, 2008 6:09 pm

    Frank,

    I can’t think of a better way of disagreeing with you than to simply repeat what I’ve already said in this thread.

    So I’ll save myself the typing and let it go.

    CAM

  12. Jay Pipes on January 28th, 2008 7:06 pm

    Curt,

    OK, thanks for pointing to your services and explaining things; that is appreciated.

    Re: whether the company I mentioned above uses triggers, SPs, and declarative integrity, I’m not sure at all. I can inquire, though.

    Cheers,

    Jay

    p.s. It might assist others looking to answer your post question if you gave us an example of something you’re thinking about. Are you talking about financials? ERPs? Something else?

  13. Curt Monash on January 28th, 2008 10:19 pm

    Jay,

    There are two intertwined sets of issues here:

    1. Does a DBMS have what it takes to support traditional, complex OLTP applications?
    2. Does it matter whether a DBMS has what it takes to support traditional, complex OLTP applications?

    If I had to pick, I’d say the second one is more interesting, even though in this thread I’m focusing somewhat more on the first.

    Obviously, an extreme “No” to #2 is ridiculous, but even so opinions can differ a lot. As they used to say at the Kennedy School, where you stand depends upon where you sit.

    As for what would be a complex OLTP app — well, ERP and airline reservations are probably the top two examples. The latter is what Dan Weinreb posted about in another thread. The former is why I say that anything with SAP certification — something MySQL so far has been unable to achieve — is ipso facto capable of handling them. But I’m trying not to be that simplistic.

    Besides, SAP itself doesn’t really believe in complex relational data modeling any more for its new application development. I’ve posted about that several times, I think.

    CAM

  14. John Sequeira on January 28th, 2008 11:21 pm

    I don’t have the detailed first-hand knowledge you’re looking for, but for some second hand info check out ’12 Days of Scale out’

    http://www.mysql.com/news-and-events/?year=2007

    And less recently, Sabre is the highest profile non-wikipedia,-yahoo,or-google mysql user that I know of.
    http://www.mysql.com/news-and-events/press-release/release_2003_33.html

    You should be able to find case studies w/stats for Sabre
    Holdings online, because I believe their conversion from Oracle was celebrated at more than one mysql/oss user conferences. Note that they used Golden Gate for replication when they decloaked, but I still count that as a huge MySQL success.

    Facebook and Youtube (and yahoo/google) are mammoth mysql deployments but I believe sizing details have not been published. I have seen/heard several conference video/podcasts on these but they’ve been cagey on the details.

  15. John Sequeira on January 28th, 2008 11:29 pm

    Oh yeah — dotlrn.org is an LMS that runs on postgresql and is heavy-duty transactional using triggers/DRI/etc.

    I know they have multiple 10K+ user deployments, though you’d could get better stats by reaching out on their bboards.

  16. Curt Monash on January 29th, 2008 2:37 am

    John,

    That dotlrn.org one looks interesting. Thanks!

    I’ll add them to the tags on the original post and see if anybody magically shows up on reindexing. ;)

  17. PaulM on January 29th, 2008 3:31 am

    Interesting set of statistics from various comments.

    The 8 billion transactions per day is some kind of telco or financial company. If it was a brokerage given the data, the 7000 or so mysql instances would equate to one instance per equity/stock.
    At least seeing what other vendors with large web traffic have done, it would suggest you would shard on each equity. If it was a telco, shard on each district/street code. Then aggregrate into another database.

    In terms of the usage outlined by Jay none of our mysql customers come close, though I will check where they do rank. The day job I am talking about…

    Have Fun
    Paul

  18. Curt Monash on January 29th, 2008 6:07 pm

    Paul,

    Your comment on brokerage reminds me of the fantasy stock market http://www.blogshares.com Data corruption is so common they have a UI feature that adds up total stockholdings, compares to shares outstanding, and notes the difference between the two figures that of course should ALWAYS be identical.

    If ACID is what one is looking for, Blogshares has pH around 12. Over a serious DBMS, it would take real effort to build an app that bad.

    I greatly doubt the Blogshares folks used InnoDB.

    CAM

  19. Log Buffer #82: a Carnival of the Vanities for DBAs on February 1st, 2008 1:53 pm

    [...] is perhaps what Curt Monash of DBMS2 does when he groups it (and others) with FileMaker. He asks, “What hard-core transactional applications have actually been built in MySQL, PostgreSQL, Ente…. And he gets plenty of [...]

  20. DBNews 2007 #4 | PettiNix on February 3rd, 2008 7:04 am

    [...] paio di letture da consigliare questa settimana, la prima del solito DBMS2 dal titolo ” What hard-core transactional applications have actually been built in MySQL, [...]

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.