I have the enviable task of researching online game and virtual world technology. My first interview, quite naturally, was with the lead developers of a game I actually play – Guild Wars. The overview is in another post; that may provide context for this one, which focuses on the database technology. (I also did a short post just on the implications for Guild Wars players.) It also has a brief description of what Guild Wars is – namely, a MMORPG (Massively MultiPlayer Role-Playing Game) with the unusual feature that most of the game world is instanced rather than utterly shared.
First, some scope. ArenaNet (Guild Wars’ developer, now a subsidiary of NCsoft) runs Microsoft SQL Server, mainly Enterprise Edition, having just switched to 2005 4 months ago. They run 1500-2500 transactions/second all day, spiking up to 5000 in their busiest periods. They have no full-time DBA, and when the developers started this project they didn’t know SQL. They’ve only had one major SQL Server failure in the 2+ years the game has been running, and that was (like most of their bugs) a network driver problem more than an issue with the core system.
As for what’s going on — there are a few different kinds of database things that happen in an instanced MMORPG.
- Game characters acquire loot; i.e., there are inventory transactions. These are commonly one-sided. Goods are created pretty much out of thin air – or rather thin electricity – and so there’s no notion of matching a deletion from stock against consumption in manufacture or anything like that. Similarly, they buy and sell at merchants and traders and so on. One character carries around 60 different items, each of which is a combination of several tokens or attributes chosen from a finite list.
- Characters evolve. They gain power. The portion of the game map they can see increases. They experience milestone events in the game. Etc.
- There are temporary changes to the game world as characters explore areas. These need to be maintained as state while the character is online and in the specific area; but because the game is instanced, they don’t need to be persisted beyond that (with minor exceptions).
- There are a variety of secondary systems with ordinary transactions, such as guild rosters and tournament standings. Based on their functionality, it would seem these have extremely simple schemas.
In some MMORPGs, inventory is handled in the obvious way – each item has an associated record. But Guild Wars does things differently. Everything about a character is one big BLOB (Binary Large OBject), typically 10K-30K in size. (One component is the whole game map, or at least a record of which pixels the character has opened up.) More precisely, there’s a character BLOB and an account BLOB. BLOBs are updated as the game progresses, then saved every few minutes.
Actually, if there are are 150,000+ transactions per minute, and each character generates a transaction every few minutes, that suggests there are ½ million or so characters online at once, peaking at 1 million. Based on what I know about gameplay and so on, I think that figure is too high by a factor of 10 or so. I can’t immediately explain the discrepancy.
Of course, there’s almost no simpler kind of transaction than banging a BLOB into a database. Even so, that’s a lot of BLOB-banging. So between the game server and the database backends, there are custom cache servers. The game servers send messages about various kinds of structured game-object transactions to the cache servers; the cache servers then bang updated BLOBs into the database. These cache servers are extremely stable, staying up 150-200+ days at a time.
ArenaNet is, on the whole, quite responsive to player requests for upgrades. But in one regard they’ve consistently disappointed people – they have omitted introducing an ingame auction house, something competitive MMORPG makers seem to be able to build. Even after the interview, I’m not 100% clear on the reasons, but I think they mainly boil down to this – an auction house would be by far the most complex transactional system they’ve implemented. And given that the inventory isn’t currently handled on a record-by-record basis, it pretty much would have to be built from scratch. That said, while they gave me no hint of this — when they reexamine the issue for Guild Wars 2, I bet they’ll realize it’s not as hard as they now think.
As for other database futures – just taking account (as oppose to character) inventory from 20 to 80 slots required database scale-out. More database scale-out in the future is likely, although they haven’t firmly decided on that yet. ArenaNet seems very happy with SQL Server, and seems unlikely to fix what isn’t broken; besides, they’re based in Bellevue. (By way of contrast, Sony Online is in the process of moving Everquest from Oracle to Enterprise DB. I hope to interview them in the future.) ArenaNet doesn’t seem to be considering memory-centric technology, although that seems an obvious choice for caching; in particular, StreamBase seems to be getting a little traction in the game/virtual world market. (More on that later if the StreamBase folks help me get some interviews.)
- A mid-2009 post on the database technology of Lord of the Rings Online and other MMOs.