Recently I've been thinking a lot about the kinds of challenges games present you with. I'm not happy with the way we think about challenges these days, the various clumsy categorizations and forced pigeonholes. Originally, this was a bit of a rant about that, but I'd much prefer to talk about ways of making challenges interesting. This applies to both video games and tabletop games.
Let's start with the idea that combat has a certain complexity. Complexity is a combination of how hard it is to predict, and the importance of predicting.
For example, at first glance "Superman vs 50 thugs" seems complex because there are a lot of moving pieces. But the truth is that it's very simple: no matter how well the thugs do, the outcome is going to be the same. It may be hard to predict exactly what each thug will roll, but it's easy to predict what the outcome will be.
Most games focus on bringing the enemies into the same power category as the players. Superman doesn't fight thugs, he fights Bizarro Superman, or Brainiac, or something. This works well, but only as a basic patch: it doesn't add to the challenge, it just makes the challenge a challenge.
Sometimes a game will bring the players down to the enemies' level. Superman fights 50 thugs, sure, but beneath Lex Luthor's Red Sun Lamp, which makes him very weak. This is generally not recommended as it negates a lot of the player's built-up agency (statistical character development). However, if the player chooses to have such a weakness, abusing that weakness is fair game, since the player's agency is being expressed rather than negated. Still, it's often quite difficult to work into a game without being irritating both in terms of play and emotion.
A third option, one that doesn't get enough attention, is to change the goal condition. Superman fights 50 thugs, but the 50 thugs are all brainwashed cops. There's nothing in the rules that says you can't tear them apart and burn them to ashes, but you're going to want to limit yourself anyway. Or there are 50 thugs, and you have to beat them all in 5 rounds or a bomb goes off. Or any number of other goal-shifts.
Most tabletop games have two distinct categories of challenge: combat and everything else. It's pretty rare for a game to have rules for goal-shifting within combat: normally shifted goals fall under "everything else" and are resolved using the much simpler and less interesting rules for that sort of business. It's worth considering goal-shifted combat explicitly, as it takes the dense combat rules and puts an interesting twist on them.
I need to stress that this really isn't something you can retrofit. D&D rules technically allow for GMs to make a horde out of brainwashed cops, or give you only a few rounds to beat all the enemies, but the rules aren't really structured correctly to make those challenges complex. In order to shift goals in nearly every combat, you need to approach combat from the perspective of achieving those goals, rather than achieving the kill-or-be-killed goal. At the very least, you need your heroes to have combat capabilities that take radically different lengths of time.
The other problem is that goal-shifting feels a little bit cheaty at first. If you goal shift just once in a while, it feels a bit epic, that's fun. But if you goal shift all the time, it can feel cheap. This is simply because we're used to it feeling epic, so if we encounter it all the time, it might feel like we're trying to be epic all the time. However, if we build a system specifically to do this, and make it clear that every combat is goal-shifted, it can simply become part of the tactical terrain.
This is very similar to adding externalities, such as "if you kill these guys, that city will hate you". In fact, it's so similar, we're going to pretend it's the same.
Challenges also involve some amount of time-shifting. Or, um, risk-pushing. I need a better term. Let's call it "risk projection".
Basically, as a player fights, he uses up resources. Some of these resources are almost instantly refreshed, such as the basic "your turn" resource. Others are refreshed fairly fast, such as HP in a first person shooter. Some take some time to refresh, such as rocket ammo in an FPS or HP in a tabletop RPG. Some require significant long-term effort to refresh, such as potions or levels.
You don't need to put absolutely everything you ever see into this framework, but the guideline here is that players will want to divide up the risk they face so that it is least risky over the long run. This means they will usually tend to lean on short-refresh things as much as possible, unless they know that a refresh point is coming up for a particular other resource or there is a threat to another, more expensive resource (like HP).
Randomness is critical to this kind of system, as the whole idea eventually boils down to whether a player is willing to risk the slightly weaker "basic stab" or whether he decides he needs the stronger, only-twice-daily "kick to the groin" attack. I know it makes no sense for that attack to be limited to twice a day, but that's the way it works in our stupid little game. And D&D 4.
You can also project risk in dimensions other than time. For example, you might choose to send your army to attack a stronghold, or you might decide to attack it yourself with minimal support. One puts the risk on you, the other puts the risk on your army. It's the same basic idea: allocate risk where it is least likely to kill you in the long run.
The idea of risk projection is one that is usually all but ignored, especially in modern video games. They restore you to complete health and ammo after every fight. I think this is one reason why modern video games feel so incredibly shallow to me.
Risk projection also functions backwards to a degree, when players decide what they will bring on a quest or what they'll take on a level up. Instead of trying to preserve resources until they need them, they instead try to figure out which resources they will need. While nearly every game does this to some extent, in most there is a clear dominant strategy for any given set-up, so it hardly counts.
It's also worth mentioning that risk projection (both forwards and backwards) integrates quickly and easily with secondary play types. For example, equipping yourself and using up equipment (did you bring enough healing potions?). Making political allies and enemies based on the side effects of your fights (did you burn the town down killing those goblins?).
When determining secondary play types, I usually try to integrate them into both the primary play (this sword does more damage than that sword) and the risk projection (this sword breaks on a botch, that sword doesn't).
Some games also use some kind of elemental sting. If the player is a fire-based character, they'll hit you with an ice level where you take huge amounts of damage, or they send people who are immune to fire against you, or whatever.
This is just an extension of the classics: sending the fast character up against the tank, sending the slow character up against an archer, sending the archer against a horde, etc, etc. In any rule set, you'll want tradeoffs and strengths and weaknesses. If you can't identify very many of them, your system just isn't good enough.
Taking the "elemental route" is a quick way to establish strengths and weaknesses. I'm not sure, but I think I feel that it is a cheap way to do it. Anyway, it is fundamentally the same kind of thing as just making monsters with the right stats to go up against characters with particular stats.
What I mean is that you can certainly challenge the ice-based players by sticking them over a lava pit and making them fight magman. But you can get the same kind of challenge by poisoning them and making them fight a jailer, or by making them fight a troll during an earthquake, or any of a hundred other situations. There is nothing statistically important about it being elemental. It's just an easy way to do it.
Anyway, those are the basics of challenges, in my mind.