Two years ago I wrote about the database technology of Guild Wars. Not coincidentally, Guild Wars was the MMO RPG (Massively Multiplayer Online Role-Playing Game) I then played. I had the chance to interview Guild Wars’ lead developers. While much else they had to say was impressive, Guild Wars’ database architecture was — er, it was rather mind-boggling.
Since then, Linda and I have taken to playing Lord of the Rings Online, commonly known as LOTRO, developed by Turbine, Inc.. I haven’t had the chance to interview any Turbine folks, despite repeated requests. But from afar, it would seem that Turbine’s technology choices leave quite a bit to be desired, in enterprise-like IT areas such as authentication, database management, and storage.
LOTRO and other Turbine games commonly are down, for scheduled maintenance or in some cases otherwise. There is scheduled multi-hour downtime to start many weeks. There are fairly frequent server restarts in addition to that. Lag and congestion are frequent. And so on and so forth. By way of contrast, Guild Wars very rarely goes down, and other technical difficulties are less common as well. Reliability is a key design goal for Guild Wars’ developers, and in my opinion they’ve achieved it.
Some of the reasons for Turbine’s difficulties seem related to the stresses of MMOs — e.g., they’re probably due to the problems caused by having many fictional characters moving through the same fictional space at once, with graphical detail much richer than Guild Wars’. But a couple of head-scratchers make me really wonder about how Turbine manages data. For example, LOTRO’s “log-in servers” are down a lot — sometimes they’re down across all Turbine games — and congested at other times. It doesn’t seem as if this is just a ruse to ration access to the main game.
Note: I don’t actually know whether LOTRO’s authentication goes through a DBMS or, say, an LDAP store. But it seems as if the core of the authentication system is the LOTRO forums (i.e., message boards), and such things are usually DBMS-based.
That’s been going on for as long as I’ve played the LOTRO (but it seems to be getting worse with each new release). Another kind of glitch finally leads me to make this post. To wit:
In a game such as LOTRO, characters have gear, such as pieces of armor. Items can be created (“crafted”), carried around, worn, bought, or sold. These have various stats — “armor level” and the like. Every piece of the same kind has the same stats, as well as the same name, appearance, and so on.* One would presume that behind the scenes, each kind of equipment has a unique SKU-like ID, stored in a database of some kind, which is then looked up as needed. If stats for an item suddenly and erroneously change, one would assume:
- It’s a database error.
- Some human at Turbine made an unintended change to a database record (because spontaneous database corruption Should Not Happen).
- The error, once detected, can be changed by a human in a fraction of a minute.
*Plus in some cases a few custom attributes — dye color, crafter’s name, custom moniker (a short text string) the crafter gave to the item.
But that set of assumptions turns out to be incorrect. I detected such an error, and contacted a game master — i.e., help desk person. I was told:
- It’s a known error.
- The fix won’t be slipstreamed.
- The fix also won’t be in a routine weekly patch.
Rather, the fix has to wait for the next significant release — a new “book” — of the kind that happens once a quarter at best.
Once again, the mind boggles.
And in other cases, performance is just sluggish. There are numerous actions in the game that seem to be database queries, and those can often take 10 or so seconds to resolve. Efficient and reliable database technology just does not seem to be a priority for Turbine. And I actually have one piece of anecdotal evidence directly supporting that theory, which I shall not embarrass my source by spelling out …
I don’t think Turbine is alone in this either. Yes, Linden Labs/Second Life uses Streambase (I don’t know what the actual DBMS is), and Sony Online uses EnterpriseDB. Somebody also told me they expected their technology to be used in the forthcoming Knights of the Old Republic (KOTOR) MMO, although I’ve forgotten who. But on the whole, the MMO gaming industry doesn’t take database technology as seriously as I would wish.
Edit: Commenters below point out that Second Life and World of Warcraft both run on MySQL. Thanks!