Candy Tests

I am a huge fan of playtesting. I playtest everything I build. Every game I make has a playtest phase before I even think about releasing it, and these playtests always catch issues that didn't occur to me. Every time.

Of course, there's two weaknesses to playtesting that you have to keep in mind at all times.

The first is that playtesters aren't (usually) testing the game for the length of time that your fans are actually going to play it. This means your more long-term play usually goes largely untested.

The second problem is that playtesters are subject to what I call the "Pepsi Challenge" fallacy.

The Pepsi Challenge was to determine which soft drink tasted better: Pepsi or Coke. Random testers nearly universally chose Pepsi. Why? Because Pepsi has a higher sugar content. It's a sweeter drink.

But it turns out that over the long run, Pepsi is less pleasing. Coke still wins out for the long-term drinkers, because it isn't too sweet.

What this means in terms of game playtesting is that testers will often choose whatever mode makes them feel stronger. However, a slope of challenge is what makes the game have staying power. So the testers will choose the variant that is actually worse for the long-term appeal of your game.

Basically, what you have to remember when using testers is pretty simple:

Watch them. Don't take what they say too straight: they aren't designers, they aren't testing the long-term play of the game, and they're heading for a Pepsi Challenge Fallacy.

What matters isn't what they say they prefer: it's what their play says. If they struggle to play, you need to polish that. But if they test two variants and have no trouble with either, the better variant is not necessarily the one they prefer.

Of course, some people are naturally immune to Pepsi. These people make great playtesters, but often don't represent your target audience very well. :)