June 9, 2007

The database technology of Guild Wars

I have the enviable task of researching online game and virtual world technology. My first interview, quite naturally, was with the lead developers of a game I actually play – Guild Wars. The overview is in another post; that may provide context for this one, which focuses on the database technology. (I also did a short post just on the implications for Guild Wars players.) It also has a brief description of what Guild Wars is – namely, a MMORPG (Massively MultiPlayer Role-Playing Game) with the unusual feature that most of the game world is instanced rather than utterly shared.

First, some scope. ArenaNet (Guild Wars’ developer, now a subsidiary of NCsoft) runs Microsoft SQL Server, mainly Enterprise Edition, having just switched to 2005 4 months ago. They run 1500-2500 transactions/second all day, spiking up to 5000 in their busiest periods. They have no full-time DBA, and when the developers started this project they didn’t know SQL. They’ve only had one major SQL Server failure in the 2+ years the game has been running, and that was (like most of their bugs) a network driver problem more than an issue with the core system.

As for what’s going on — there are a few different kinds of database things that happen in an instanced MMORPG.

In some MMORPGs, inventory is handled in the obvious way – each item has an associated record. But Guild Wars does things differently. Everything about a character is one big BLOB (Binary Large OBject), typically 10K-30K in size. (One component is the whole game map, or at least a record of which pixels the character has opened up.) More precisely, there’s a character BLOB and an account BLOB. BLOBs are updated as the game progresses, then saved every few minutes.

Actually, if there are are 150,000+ transactions per minute, and each character generates a transaction every few minutes, that suggests there are ½ million or so characters online at once, peaking at 1 million. Based on what I know about gameplay and so on, I think that figure is too high by a factor of 10 or so. I can’t immediately explain the discrepancy.

Of course, there’s almost no simpler kind of transaction than banging a BLOB into a database. Even so, that’s a lot of BLOB-banging. So between the game server and the database backends, there are custom cache servers. The game servers send messages about various kinds of structured game-object transactions to the cache servers; the cache servers then bang updated BLOBs into the database. These cache servers are extremely stable, staying up 150-200+ days at a time.

ArenaNet is, on the whole, quite responsive to player requests for upgrades. But in one regard they’ve consistently disappointed people – they have omitted introducing an ingame auction house, something competitive MMORPG makers seem to be able to build. Even after the interview, I’m not 100% clear on the reasons, but I think they mainly boil down to this – an auction house would be by far the most complex transactional system they’ve implemented. And given that the inventory isn’t currently handled on a record-by-record basis, it pretty much would have to be built from scratch. That said, while they gave me no hint of this — when they reexamine the issue for Guild Wars 2, I bet they’ll realize it’s not as hard as they now think.

As for other database futures – just taking account (as oppose to character) inventory from 20 to 80 slots required database scale-out. More database scale-out in the future is likely, although they haven’t firmly decided on that yet. ArenaNet seems very happy with SQL Server, and seems unlikely to fix what isn’t broken; besides, they’re based in Bellevue. (By way of contrast, Sony Online is in the process of moving Everquest from Oracle to Enterprise DB. I hope to interview them in the future.) ArenaNet doesn’t seem to be considering memory-centric technology, although that seems an obvious choice for caching; in particular, StreamBase seems to be getting a little traction in the game/virtual world market. (More on that later if the StreamBase folks help me get some interviews.)

Related link

Comments

14 Responses to “The database technology of Guild Wars”

  1. coridan on June 10th, 2007 8:30 am

    Great articles very well thought out and written… Thanks for taking the time and looking forward to more of your work in the futue

  2. Andrew on June 12th, 2007 4:52 am

    Very interesting, good read. I guess I never thought about it, but I never thought about commercial games companies using SQL based databases, thought they’d use an in-house thing. Makes sense though if I think about it.

  3. deviceZero » Technical Guild Wars Facts on June 12th, 2007 10:43 am

    [...] technical facts about Guild Wars from these three [...]

  4. Bour Gie on June 12th, 2007 1:02 pm

    Thanks for the read. I’ve been playing GW for a couple of years and, as a DBA, have wondered about the technology behind the scenes. I’m surprised about the BLOBs, given some of the limitations in character gear, but hey, whatever works, right? 150k transactions a minute…. that could be right. Assuming 10% are online at any point in time, and half of those are out killing critters, that’s only 1 person killing 1 critter in 1 minute. It’s hard to run for 1 minute without running into a group of critters, so that seems plausible. It could be higher, actually, given that there are people that “solo” or “farm” areas, trying to kill everything by themselves as quickly as possible.

    Bour Gie (L20 Ranger/Ritualist)

  5. Curt Monash on June 13th, 2007 1:27 am

    No. There’s definitely a problem with my numbers, or my understanding. The key point here is that there isn’t automatically a transaction every time a critter is killed.

    And of the 1-2 million GW players total, not all that many of them are online and actively killing things AT ONCE.

  6. Stephanus Aurelius on June 13th, 2007 2:15 am

    I’m kind of curious as to if instancing was also one of their solutions to deal with possible
    transaction collisions. Curt do you know if collisions are minimized due to instancing or not?

  7. Curt Monash on June 13th, 2007 5:22 pm

    I’m sorry, Stephanus, but I don’t understand the question. I’ve never played a persistant/non-instanced MMORPG, but don’t see where collisions would come in in either case.

  8. Stephanus Aurelius on June 15th, 2007 3:29 am

    In a persistant/non-instanced MMO there can be a heavy load on servers, which would possibly cause a
    slowdown in the rate of data transfer(killing a monster, grabbing loot, buying/selling/trading, etc.)
    or even data collision inwhich the transaction is not completed. I have experienced this in Final
    Fantasy XI, but not so much in Guild Wars. So, I was wondering if instancing played a roll in
    minimizing these types of occurrences.

  9. Curt Monash on June 15th, 2007 7:58 am

    How many players share a server/slice in a persistent MMO? Could Guild Wars’ BLOB strategy even work?

    The developers did say that the BLOB strategy helped with transaction integrity, against some exploits to deliberately crash the system mid-trade and hence duplicate items.

    That said — it has to be possible to maintain transaction integrity in a persistent world too. We’re not talking THAT many individual transactions, although again it depends on the number of players per server.

  10. Log Buffer #49: a Carnival of the Vanities for DBAs « Coskans Approach to Oracle on June 15th, 2007 8:50 am

    [...] views (DMV) of SQL Server 2005. Curt Monash wrote about how you can use MSSQL Server 2005 behind a Massively MultiPlayer Role-Playing Game  [...]

  11. DBMS2 — DataBase Management System Services » Blog Archive » Thoughts on database management in role-playing games on April 25th, 2008 12:05 am

    [...] a research project on the IT-like technology of games and virtual worlds, especially MMORPGs. My three recent posts on Guild Wars attracted considerable attention in GW’s community, and elicited [...]

  12. NoSQL? | DBMS2 -- DataBase Management System Services on December 10th, 2009 3:02 am

    [...] last point is not a joke. One of the weirder database architectures I know of is the one underlying Guild Wars. Its developer — a brilliantly impressive guy — told me flat-out that he learned in [...]

  13. – Reducing The Costs Of SQL Joins, Part 1: Common Relational Database Performance Workarounds on September 6th, 2012 1:14 am

    [...] blobs can start out as being unassuming and small, but can quickly become a significant problem. Kurt Monash at dbms2 provided a good example of the use of Blobs for object storage in the context of the Massively [...]

  14. The worst database developers in the world? | DBMS 2 : DataBase Management System Services on April 16th, 2014 2:45 am

    [...] in the world, they’re at least on the short list for consideration. The makers of Guild Wars didn’t even try to have decent database functionality. A decade later, when they introduced Guild Wars 2, the database-oriented functionality (auction [...]

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.