June 14, 2009

MMO games are still screwed up in their database technology

Two years ago I wrote about the database technology of Guild Wars. Not coincidentally, Guild Wars was the MMO RPG (Massively Multiplayer Online Role-Playing Game) I then played. I had the chance to interview Guild Wars’ lead developers. While much else they had to say was impressive, Guild Wars’ database architecture was — er, it was rather mind-boggling.

Since then, Linda and I have taken to playing Lord of the Rings Online, commonly known as LOTRO, developed by Turbine, Inc.. I haven’t had the chance to interview any Turbine folks, despite repeated requests. But from afar, it would seem that Turbine’s technology choices leave quite a bit to be desired, in enterprise-like IT areas such as authentication, database management, and storage.

LOTRO and other Turbine games commonly are down, for scheduled maintenance or in some cases otherwise. There is scheduled multi-hour downtime to start many weeks. There are fairly frequent server restarts in addition to that. Lag and congestion are frequent. And so on and so forth. By way of contrast, Guild Wars very rarely goes down, and other technical difficulties are less common as well. Reliability is a key design goal for Guild Wars’ developers, and in my opinion they’ve achieved it.

Some of the reasons for Turbine’s difficulties seem related to the stresses of MMOs — e.g., they’re probably due to the problems caused by having many fictional characters moving through the same fictional space at once, with graphical detail much richer than Guild Wars’. But a couple of head-scratchers make me really wonder about how Turbine manages data. For example, LOTRO’s “log-in servers” are down a lot — sometimes they’re down across all Turbine games — and congested at other times. It doesn’t seem as if this is just a ruse to ration access to the main game.

Note: I don’t actually know whether LOTRO’s authentication goes through a DBMS or, say, an LDAP store. But it seems as if the core of the authentication system is the LOTRO forums (i.e., message boards), and such things are usually DBMS-based.

That’s been going on for as long as I’ve played LOTRO (but it seems to be getting worse with each new release). Another kind of glitch finally leads me to make this post. To wit:

In a game such as LOTRO, characters have gear, such as pieces of armor. Items can be created (“crafted”), carried around, worn, bought, or sold. These have various stats — “armor level” and the like. Every piece of the same kind has the same stats, as well as the same name, appearance, and so on.* One would presume that behind the scenes, each kind of equipment has a unique SKU-like ID, stored in a database of some kind, which is then looked up as needed. If stats for an item suddenly and erroneously change, one would assume:

*Plus in some cases a few custom attributes — dye color, crafter’s name, custom moniker (a short text string) the crafter gave to the item.

But that set of assumptions turns out to be incorrect. I detected such an error, and contacted a game master — i.e., help desk person. I was told:

Rather, the fix has to wait for the next significant release — a new “book” — of the kind that happens once a quarter at best.

Once again, the mind boggles.

And in other cases, performance is just sluggish.  There are numerous actions in the game that seem to be database queries, and those can often take 10 or so seconds to resolve.  Efficient and reliable database technology just does not seem to be a priority for Turbine.  And I actually have one piece of anecdotal evidence directly supporting that theory, which I shall not embarrass my source by spelling out …

I don’t think Turbine is alone in this either. Yes, Linden Labs/Second Life uses Streambase (I don’t know what the actual DBMS is), and Sony Online uses EnterpriseDB. Somebody also told me they expected their technology to be used in the forthcoming Knights of the Old Republic (KOTOR) MMO, although I’ve forgotten who. But on the whole, the MMO gaming industry doesn’t take database technology as seriously as I would wish.

Edit: Commenters below point out that Second Life and World of Warcraft both run on MySQL. Thanks!

Comments

18 Responses to “MMO games are still screwed up in their database technology”

  1. Markus Perdrizat on June 14th, 2009 5:35 pm

    Not knowing anything about the underlying tech of the MMOGs you play, I instantly got reminded of this saying: If you only have a nail, every tool looks like a hammer.

    If you’re a database guru, every performance problem looks like wrong usage of a database (or database misconfiguration, depending on how much insight you have).

  2. Curt Monash on June 14th, 2009 5:51 pm

    I thought I explicitly made reference to performance problems that had nothing to do with database technology.

    But when it’s simply a matter of listing a lot of items that meet certain criteria in a simple UI — yeah, that seems like a database matter to me.

    CAM

  3. Jerome Pineau on June 14th, 2009 6:31 pm

    Well I can tell ya Blizzard (Irvine,CA) struggles with MySQL on a daily basis 🙂

  4. Curt Monash on June 14th, 2009 9:10 pm

    Would it be appropriate to say WOW? 🙂

  5. keru on June 15th, 2009 4:00 am

    Linden Lab use mySQL.
    You can find many reference to mysql on their blog, and also on : http://www.mysql.com/customers/view/?id=699

  6. Daniel Weinreb on June 15th, 2009 7:03 am

    MMOG’s have different DBMS requirements than the classic, e.g., financial system. Low latency is absolutely critical. However, losing a very small amount of data during a failover is relatively tolerable.

    At Object Design, our ObjectStore object-oriented database system worked the classical way: all transactions must be ACID, no matter how much latency that takes. One of the top ObjectStore designers, Tim Blackman, is now working at Sun Labs on Project Darkstar, where he recently told me about the new DBMS technology he and the team are developing specifically to meet the needs of MMOG’s. Check it out:

    http://projectdarkstar.com/

    http://en.wikipedia.org/wiki/Project_Darkstar

  7. The database technology of Guild Wars | DBMS2 -- DataBase Management System Services on June 15th, 2009 8:08 am

    […] post on the database technology of Lord of the Rings Online and other MMOs. Categories: Application areas, Games and virtual worlds, Microsoft and […]

  8. Curt Monash on June 15th, 2009 9:17 am

    Dan,

    MMOs have what’s described as an “in-game economy”, with manufacturing and retail transactions resembling those of real life. Those need to be ACID for the same reasons they need to be ACID in the real world.

    CAM

  9. Daniel Weinreb on June 16th, 2009 6:16 am

    But just moving a player from point A to point A + delta does not, and that’s what has the most stringent latency requirements.

  10. Curt Monash on June 16th, 2009 6:57 am

    True, Dan, and up to a point “rubber-banding” is tolerated by players, if recent actions are lost.

    LOTRO sometimes loses 15 minutes or more of play at a time. I never experienced more than half a minute or so of loss on Guild Wars.

    Anyhow, I think the parts of LOTRO that are like OLTP transactions are sent out to different subsystems and perhaps different physical servers. And latency on them is often at the very annoying finger-tapping level.

  11. Curt Monash on June 16th, 2009 1:22 pm

    There’s a thread on the LOTRO forums based on this post, but so far it’s generated more heat than light.

    http://forums.lotro.com/showthread.php?t=273641

  12. Pete Mancini on June 16th, 2009 9:53 pm

    There is an additional dimension here that you are forgetting about. MMOs have to contend with active and malicious hackers. These dudes are active 24/7 trying to hack the game so they can win fights, have more loot, etc. A lot of rational technology fixes have to be thrown out completely because they open the door for the hackers to cause trouble. With in-game items selling for ACTUAL MONEY on sites like eBay there is plenty of motive for the hacker. The problem domain is pretty intense and beyond the normal $50/hr programmer or outsourced programming shop in Bongolesia. So highly lag free, highly secure and no data loss is one of those “choose 2 of the above” questions right now. If you can get the first two and make is so that important individual information is replicated locally that would give a means for getting the third item (albeit after the fact when the Server and Client do a sync after high traffic.) I am sure the software engineers at these MMO companies are as hard pressed to solve the problem as you are living it by playing the games.

  13. Curt Monash on June 16th, 2009 10:03 pm

    That’s actually covered in one of my articles on Guild Wars, if you follow the links.

    But classical DBMS transactions would seem to meet the test.

  14. Ben Werther on June 16th, 2009 10:25 pm

    This isn’t really DBMS related at all, but is kind of on topic–

    There is a way to mostly escape the “choose 2 of the above” (in the previous comment).
    When I was at There.com, Ken Duda (now VP of Eng at Arista) conceived of one of the most impressive pieces of technology I’ve ever come across – a massively scalable distributed physics simulation engine for MMOGs.

    The servers would run the simulation and maintain the true state of the world (allowing high security). All clients would also simulate locally based on all the information they’d received. The challenge is how to hide latency so that everything moves smoothly but the user can control their avatar interactively without waiting for a round trip back from the server. The magic was a ‘relativistic’ model where objects further away from the user would run at the server’s clock, while anything closer to the user would shift into his local extrapolated timeframe. This means that everything runs smoothly without snapping, except in the case where two users’ actions interact at close quarters in a way that conflict (and so they snap when the server tells them what really happened).

    With this model a user could play billiards in an MMOG with no local latency and fully correct collisions from the perspective of all observers. Only if two players hit balls within a few hundred milliseconds of each other would there be a chance for a momentary snap until each received each other’s event from the server. Amazing technology — and ahead of anything since as far as I’m aware.

    http://www.google.com/patents?id=ZmUOAAAAEBAJ

  15. Joe Hellerstein on June 24th, 2009 8:52 pm

    Have a look at some very interesting <a href=”http://www.cs.cornell.edu/bigreddata/games/”research out of Cornell in recent years on declarative (read: data-centric) approaches to programming game AI on a massive scale. This is consistent with a general trend toward declarative programming in research on achieving the next generation of scalable software.

  16. Joe Hellerstein on June 24th, 2009 8:53 pm

    ack.. here’s that link again to the Cornell Data-Driven Games project….

  17. Today in Cloud — GigaOM Research on October 18th, 2013 6:15 pm

    […] I way off base, or are MMO games behind the times technology-wise? Not only are some people lamenting the apparent database inadequacies of these platforms, which handle vast amounts of data, but the computing and networking aspects […]

  18. The worst database developers in the world? | DBMS 2 : DataBase Management System Services on April 16th, 2014 2:45 am

    […] real-money store, etc.) would crash for days at a time. Lord of the Rings Online evidently had multiple issues with database functionality. Now I’m playing Elder Scrolls Online, which on the whole is a great game, but which may have […]

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.