(I'm skipping a lengthy dissertation on fear. If you want to see a hint of what you're missing, here's a slightly outdated Doom III dissection. This essay turned out boring anyway. Argh!)
What's your favorite game?
Okay, unfair question. Nobody should have an answer to that. Here's a different one: what's your favorite horror game?
Maybe you're a Resident Evil kinda guy. Maybe you prefer Eternal Darkness. Maybe you have the good taste to prefer a System Shock. Maybe you picked something else - perhaps even something not normally considered a horror game, like Myst.
If you look at all the horror games you enjoy, you may notice that there is something fundamentally similar about them. Or, perhaps you can barely see the difference in the first place. Either way, you're right: there's something fundamentally similar about them.
Aside from the most primitive forms of shock, all the kinds of fear require the player to be immersed. A running gunfight is most terrifying when we're in danger. A freakshow of a cutscene is only horrifying if we care about what's going on in some way.
Immersion is the blood pumping through a horror game. The goal of every horror game is to sink the player into the setting so deeply he reacts forcefully to the mere sight of an enemy.
Breaking immersion is therefore the worst thing a game can do.
But "immersion" is a rather messy word. For example, I consider it immersion-breaking when my gun-weilding heroine can't simply blast the locked cabinet open. But others don't even notice. On the other hand, they might consider it immersion-breaking to have a health meter and ammo readout. I don't even notice.
What kind of immersion you're going for is a critical first thought a designer needs to consider before the alpha is even a twinkle in a programmer's eye.
If you want more of a "horror" game, you usually want to specialize your game to give maximum view with minimum combat effectiveness. This means a third-person view - can't shoot worth a damn, but you can see items and enemies creeping up from every direction while you're busy shooting the one spot on the wall without zombies. You're going for a slightly more "movie-ish" feel, so HUDs are out and complex stats are goners. This has the benefit of also reducing combat effectiveness, since players can no longer twink out. But you have to do very good set design, because the players are going to be interacting with it so much.
Of course, if you want more of a "terror" game, you want to specialize your game to give minimum view with maximum combat effectiveness. First person all the way: an intense, powerful view that makes it hard to look around a level (for monsters or stuff). You're going for a more "realistic" feel, so the character needs to be able to move quickly and be focused on things other than hexagonal keys and telescopes.
These aren't the only two choices, of course. But they are the most common.
Once you've designed your game to immerse the player, you have to avoid breaking that immersion on pain worse than death: unprofitability. Now, if you're doing a first-person "terror" game, you can break the setting immersion slightly without penalty. The layout doesn't have to make a terrible amount of sense, you can forego bathrooms, and so on. On the other hand, a third-person "horror" game can afford to screw up the kinetic feel a little bit more, say by making you unable to shoot out the glass between you and that hexagonal key, or by making combat somewhat uninteresting.
Their focuses are different, so their immersion is "directed" in a particular way. If the game screws up in its focus, it's very obvious. For example, Doom III screwed up. It was a terror game which screwed up the realism/kinetic feel. You couldn't use a gun and a flashlight at the same time, as the obvious example. More critical, the difficulty of the game was not immersive. (To be honest, this is because most of the game was designed to have gun OR flashlight, but ubiquitous mods made it gun AND flashlight.)
Either way you point it, there are some shared points of immersion. The most obvious is the plot. No matter what you do to immerse people in the action and puzzle sequences of the game, the plot exists separate. It doesn't require a special view type, or a combat engine. All it requires is some way of telling the player something. Cutscenes, emails, disembodied voices, s'all good.
Not every game has to have an immersive plot, but if you choose to try for one, you need to remember that you have now chosen an additional type of immersion. And now you have to struggle not to break it.
In a game whose plot is unimportant, you can make plot points that serve the gameplay but make no real sense. But not in a game where plot matters.
There are other kinds of immersion. Character development. Skill and metaphysics use. Character interactions. Team dynamics. Map building. Stealth.
Keeping things immersive gets exponentially worse with each kind of play you add. Stealth and combat? The player might be mostly stealth, mostly combat, or a mix, and those need to all be viable. Stealth, combat and teams? The player might be one of the three, mixes of two, or a combination of all three. Seven options rather than three. It gets steadily worse from there.
Each play type needs to not only be interesting and challenging, but needs to be available at every moment in every situation, including cutscenes and plots.
So, what's your favorite horror game?
I bet it focuses on just two immersions, maybe three. Probably one kind of game play and one non-gameplay shtick, such as plot or insanity or something. All the other things that could be immersive are deep in the background.
I guess the moral of this essay is: keep it simple. Every kind of play you add exponentially increases the difficulty of making the game good. Not just because you need more content, but because you need to manage immersion.