A Walk in the Dark A look in to the mind of an RPG designer

17Jan/12Off

A Plea to D&D 5th Edition Playtesters

I've been a professional computer programmer for close to 30 years, and in that time I've had my share of interaction with testers Quality Assurance. My worst experience with "testers" was when I was doing video game development, both with my Quake II add-on "PainRift" and with my Half-Life total conversion "The Opera".

Gamers are a fickle bunch... They could be pleasant and very appreciative of the wonders you create, but they can also be brutally harsh. And I can tell you that it's virtually impossible to appease all of them. Look at some of the best games in history, games like Half-Life 2 or Portal 2 that have overwhelmingly positive reviews: if you dig deep enough, you will no doubt find a handful of people that hated it with a passion. And when they hate it, oh boy they are vocal. Saying "Portal 2 sucks" isn't enough... They manage to find the time to write a veritable master's thesis full of criticism.

The thing is that criticism of that nature isn't "testing"; it's opinion. If I create a video game and someone responds "this sucks", what exactly am I to do about that? How do I make it not "suck"?

I mention all of the above because of the upcoming playtest for Dungeons and Dragons 5th Edition... or D&D Next... or D&D Whatever It's Called™. I admire WotC's ambition to try and give the power over the game's design to the fans, but this will only work if the fans give information that's actually useful.

Just to give you an idea, here are some examples of feedback I received when I released the Beta version of "The Opera":

  • "Guns are broken": Mind you, the game was released with TWENTY-TWO different guns, ranging from pea shooters (Czech Duo, Walther PPK) to handcannons (Thompson Center Contender, Desert Eagle 50AE) to long guns (Winchester 1887, Remington 12-gauge). There were upwards of 30,000 lines of code written to make all these guns work... How am I to guess which line(s) of code is "broken"?
  • "Diving sucks": One of the aspects of TO was the ability to move in special ways, such as making John Woo-like dives and rolls with guns akimbo. Granted, mod developers don't have 100% control over player movement (or at least we didn't back then) since most of that code is internal to the game engine, so it wasn't an exact science. It was unavoidable that there be problems with collisions and the player was bound to run in to problems such as getting stuck in world geometry (like under a table or in a window); each one of these situations had to be dealt with individually on a case by case basis, so telling us that diving "sucks" really doesn't help us narrow down and correct the problem.
  • "Why didn't you include the [insert new weapon here]?": This is the worst type of feedback. No matter how big you make the game, how expansive your arsenal of weapons is, it just isn't enough because one person out there wants to see their favorite gun. And it gets worse: the theme of "The Opera" was to mimic Asian action flicks in the John Woo style of gunplay, such as the movies Hard Boiled, The Killer, etc. Although we tried to include most of the weapons seen there (the Beretta 92FS was the signature weapon, the aforementioned Thompson Contender was from Hard Boiled, etc.), people still suggested weapons that made absolutely no sense. I recall several people suggesting, quite adamantly, that we include Vulcan miniguns and rocket-propelled grenades just because they were "cool". "You know what? You should have a CHAINSAW just like Ash! That would be so awesome!" Maybe I missed something here... When did Army of Darkness join the Hong Kong blood opera genre?
  • "Why didn't you include the [insert weapon here] instead of the [insert weapon here]?" Like the above, people just aren't satisfied. We included the Beretta 92FS because it was the single most common weapon in the whole HKBO genre (heck, in at least two different movies I can think of Chow Yun Fat was carrying SIX of them on his body). But Beretta makes like 50 different weapon models, each one radically different. Nobody cared that the weapon we chose fit the theme of the Hong Kong blood opera... they prefered model XXX of the Beretta because the gun is "better" in the field or more effective in combat due to increased accuracy, larger clip, less recoil, etc. I even remember one person, who took great pride in mentioning that he was a former Marine, going in to intricate technical detail over the specifications of virtually every weapon we had and commenting on its combat effectiveness and why it would be "impractical" to use a given gun in a hostile situation. I appreciate the feedback, I really do, but we're not going to turn our game in to Call of Duty: Asian Ops.

The week after we released, our forums lit up with people grilling us over what we created. We know for a fact that the majority of people liked it, but the people that liked it were nowhere near as vocal as the few that despised it with such seething hatred that they were compelled to write volumes about how bad it was. As designers, we were forced to read those long posts that assault the very thing we've spent the last two years developing... and it hurt. It demoralized a lot of us, leaving us to wonder "why are we doing this?" The hundreds of loyal fans that loved our product were drowned out by a handful of people that sought to destroy us, and it was quite devastating.

Now, once D&D WIC™ is released that level of hatred will be unavoidable, and I'm sure that the WotC forums will be virtually unreadable (I know... they're hard to read now). But none of that helps the product.

So if you want to actually help the product, provide feedback that can be actually useful and beneficial to the designers. For example:

  • Be as detailed as possible about your play experience, from beginning to end: Note that I'm not talking about mechanics or about the nitty gritty here... Tell them what it felt like to play. What was fun, what wasn't fun, what worked and what didn't.
  • Unless told to be, do not try to be an editor: Of all the types of criticism that you can receive, the so-called "grammar nazis" are the worst. They will hate your game with a burning passion just because you used or didn't use an Oxford comma. But don't worry: part of the design process in any publication is to hire a team of editors to go through the document with a fine-toothed comb. That editing cycle may not have happened yet, or may not have been completed yet, by the time you get the testing materials. So unless WotC explicitly tells you to look for such things, do not waste time with punctuation, spelling, sentence structure, etc... There are people far more qualified than you that will do that before the product hits the shelves.
  • Be as general as possible about "game math": Do not suggest to them that an attack should create 1d8 damage instead of 1d6; that's more detail than they probably want to know at this point. Telling them that the attack causes too little or too much damage would be enough, and trust them to be able to do the math far better than you could. This playtest is not a math problem; it's a playability test to ensure that the core mechanic works. 
  • Try everything, even if you don't want to: Even in 4E, there are a lot of things that exist but haven't been errata-ed simply because people don't use them. In a recent Fourthcore Deathmatch game, we discovered a problem with a Wilden racial power, which I'm imagining went unnoticed so far because, honestly, who plays as a Wilden anyway? If you want to help create the best overall product you can, be ready to try things that you may not want to. Don't like mages? Play one anyway. Only like being elves? Be a half-orc. By trying things that you don't usually do, not only do you get to experience the things you've missed out on but you can provide valuable insight as if you were a newcomer trying the game for the first time.
  • Try the unexpected: In my first game, PainRift, one of the most vicious bugs we discovered was that the game would crash violently if two rockets collided in mid air. No amount of prepared test procedures would take such an event in to consideration, and it was up to a few players mucking about and actively trying to "break" the game to figure out the reason for it. So when you're playing, don't try to do everything "by the book"; try to do things that aren't documented, that aren't mentioned everywhere. Actively look for exploits and loopholes in the game mechanic so that they can be addressed now and not become a serious problem later. An effective tester is someone who not only tries everything that's documented, but tries to do the crazy stuff that nobody's thought of yet.
  • Have fun: Proper testing can be a grueling experience, but a necessary one. Real game testers spend hours on end playing the same game over and over again, doing repetitive tasks trying to reproduce the same nuance in game. Thankfully, D&D allows for far more flexibility, and the nature of the game itself makes it so you don't have to reproduce a "bug" in the same manner as you would in game. So if the game starts to become "not fun", that's a problem worth reporting; make note of it, explain what chain of events made the game no longer fun, then do something radically different so that the game becomes fun again. Once you do that, also document what you changed in order to make the game fun. Knowing that information will allow the designers to make the game fun more consistantly and avoid the pitfalls that lead to a un-fun game.

I hope the above helps you give WotC the kind of information they need to make the next iteration of Dungeons and Dragons the best that it can be.

-=O=-

EDITED: Someone actually reminded me that the Winchester in my own game was a Winchester 1887 (which was used in Terminator 2), not a Winchester 1911. I confused that weapon with the Colt 1911s (which were used in Face/Off). I think the fact that someone went through the effort to point this out more than proves my point.

EDITED #2: And it's a Czech Duo... Not a Colt Duo... Screw it, I give up. :P

Filed under: 5E, DnD, Mechanics, RPG Comments Off
Comments (10) Trackbacks (4)
  1. I absolutely agree that feedback is often incredibly unhelpful and good playtesters have a serious job. While at first glance a professional playtester “gets paid money to play games!” they often have to play them in ways that you would never normally play a game specifically to try their darndest to flex it so hard it breaks.

    Giving the what, as well as the why, in addition to the process (the what and why right next door on either side time-wise), in further addition to a couple of suggested fixes to outline exactly what direction you think could be taken is extremely important to the playtesting process, and since I fully intend to play D&D Next I also fully intend to take advantage of this process to give the best darn feedback possible. Then again, there are people who don’t try to live eat sleep and breathe game design unlike me, so this definitely had to be said. Three cheers for this article!

  2. My only input is that when I’ve tried testing new mechanics with people I’ve played with for a long time, I still get crappy feedback.

    One suggestion: Math feedback, as in, “here is how you fix the formulas” is usually not helpful. Math feedback, as in, “[in depth mathematical analysis of system A] seems to be problematic in light of [in depth mathematical analysis of system B]” has always helped me. It also shows me where some of the crunchier aspects of my descriptions are confusing or can be exploited.

  3. I’m posting this to my playtest group as required reading.

    I especially like the bit about “try everything” – good advice even for non-playtesting, but better still in this early phase.

  4. This is great. As someone who just got a lot of playtest feedback, I think you’re right on the money.

    With something like a roleplaying game, you may even know a bit about the math behind the choices. If you find that something is different than your progression chart, like a monster having different damage or HP than it’s level would suggest, don’t correct it. Chances are high the designer has the same or better access to these numbers. This was likely a choice.

    If you find this is causing a problem in play, speak to that. “We found this combat ran long. The orc has a lot of HP, and seemed more difficult to kill than other monsters of this level.” Is a lot more useful than “Th orc should have 53 hp.”

  5. In an open test environment of this scale I don’t think WotC need a lot of feedback from people who don’t normally play Wizards, playing Wizards and then explaining how to make them less like Wizards. If no-one wants to play Wilden then they ca afford to be less vigorously tested.

  6. Agreed. Thankfully, they’ve said that much of their user feedback will come by way of user surveys. This will allow Wizards to contain the feedback to those areas they are currently working on and will hopefully avoid the “XYZ Sucks type of feedback”. I also think they have a really solid group of designers on the project whos final say will be what matters. Ultimately, whatever WotC does, the players will likely be more accepting (the reasonable ones anyway) as they will feel they’ve had a hand in its creation even if only a small one.

  7. Good points, and I think Wizards understands this. The Friends and Family round of playtesting includes a number of people that have the playtester beholder icons on DDI accounts. I’m one of them (and thus may be biased) and in my opinion these are very good playtesters that understand how to provide valuable feedback. Several of the playtesters regularly receive (as well as provide) playtest feedback and know how to break things down into useful bits and how to say more than “didn’t work” and instead help designers know the underlying issue.

  8. I remember The Opera. I didn’t get to play the mod (since I didn’t have Half-Life), but the various stories at The Marked Bullet and on the RedeemedSoft forums (both of which are unfortunately down now) ended up drawing me in, as I was a huge Heroic Bloodshed fan back then (still am, to some degree). They even had a nice little guide to creating HKBO fanfiction, which I probably still have saved on my machine.

    You’ve brought up some very good points and I hope that Wizards takes them to mind when picking the brains of the fans. While I’m personally not too optimistic about the new edition pleasing EVERYONE (since every edition lends itself to a different play-style and we as gamers and human beings are prone to nitpicking and disagreement), my general tack, as with most things of this nature, is to simply sit back and see what develops before making a decision.

    • FYI: I haven’t checked The Marked Bullet, but the RedeemedSoft site is visible pretty well on the Wayback Machine (archive.org).

      I still have the server hard drives around here somewhere… Wonder if they still work after 10 years… :P

  9. Adoro me mostrar peladinha na web cam