Sunday, November 12, 2006

Candy Tests

I am a huge fan of playtesting. I playtest everything I build. Every game I make has a playtest phase before I even think about releasing it, and these playtests always catch issues that didn't occur to me. Every time.

Of course, there's two weaknesses to playtesting that you have to keep in mind at all times.

The first is that playtesters aren't (usually) testing the game for the length of time that your fans are actually going to play it. This means your more long-term play usually goes largely untested.

The second problem is that playtesters are subject to what I call the "Pepsi Challenge" fallacy.

The Pepsi Challenge was to determine which soft drink tasted better: Pepsi or Coke. Random testers nearly universally chose Pepsi. Why? Because Pepsi has a higher sugar content. It's a sweeter drink.

But it turns out that over the long run, Pepsi is less pleasing. Coke still wins out for the long-term drinkers, because it isn't too sweet.

What this means in terms of game playtesting is that testers will often choose whatever mode makes them feel stronger. However, a slope of challenge is what makes the game have staying power. So the testers will choose the variant that is actually worse for the long-term appeal of your game.

Basically, what you have to remember when using testers is pretty simple:

Watch them. Don't take what they say too straight: they aren't designers, they aren't testing the long-term play of the game, and they're heading for a Pepsi Challenge Fallacy.

What matters isn't what they say they prefer: it's what their play says. If they struggle to play, you need to polish that. But if they test two variants and have no trouble with either, the better variant is not necessarily the one they prefer.

Of course, some people are naturally immune to Pepsi. These people make great playtesters, but often don't represent your target audience very well. :)


Patrick said...

Well put, maybe the notion of a "tissue tester" needs revision. Do you think building a long-term session relationship (that is, enough sessions to gauge long-term play) is a better alternative?

Craig Perko said...

Sure, it's better, but it's still not going to approach the tens or hundreds of hours your fans will put in. You'll still have the same problem, just a little less of it.