Back when I first started running games, I was a bit of a control freak. I think most GMs start that way, especially if they're still in their mid teens when they start.
But it didn't take me too long (half a decade? Gah...) before I realized the joys of not being a control freak.
As time progressed, I began to define my own ideas more and more loosely, then just adapting to whatever the players put into the game. If you've done this, you've probably found the same thing I did: more cannibals make better soup. Or something.
Basically, it just comes down to time. The more freedom you give them, the more time they spend. I feel vaguely guilty when I find out that some of my players spent literally thirty hours each in one week on one game, when I spent about ten on the game in total... but guilty or not, the end result speaks for itself: the story is more detailed, more varied, more interesting, and more personalized than I could have possibly created even spending thirty hours a week on it.
There's something about people creating stories together. Maybe it's just that everyone has different areas they're good at, and by combining you can pick the best talent at the best moment for a more universally good story. Maybe it's sheer dint of time and effort. Maybe it's the way that players will feed off each other more and more excitedly the more time you give them together.
Likely, it's all of those factors and others combined.
But the end result is simple:
Stop writing your games!
All o' you combined are more talented than you are alone, and players like having some measure of control over the story. Most of them might feel nervous ("I can't tell a good story!"), but once they've gotten past stage one they contribute some of the most astonishing gems. Something to do with the fact that they don't look at storytelling the same way, I think.
I'm not saying you can't have a story. Hell, every game I run is seeded with plot elements and metaphysics, and even if I waltz away for a month and let it run itself, I find that the players pretty much act out the story I had planned. Except with a level of detail and breadth of quality that astound.
Now, you can try to do this kind of thing with D&D. Except you'll find that the only way to control the story in D&D is by the PCs having huge amounts of power, which basically turns the whole thing into a terrible wish-fulfillment thing that leaves all the players glaring at each other.
Like any kind of play, if the play isn't deep, the play doesn't entertain. You've got to have some method of controlling story elements. Something the players can sink their teeth into while they work with the story. Something that doesn't scare newbies, but is deep enough to entertain the experts. Simultaneously.
Most GMs don't have a clue what I'm talking about. Some of them - probably many of the people who read this post - are thinking in terms of Nobilis or hazily-remembered reviews of "Life with Master" or PrimeTime. A few may have actually played one of these games, or at least seen a rule book.
Well, I'm'a gonna write up a big "playing narrative" guide, because I'm sitting on all of those and a dozen more. And because I'm completely exhausted and burned out. Obviously, what I need to do is write up a post on how to do interactive narratives.
I hope you'll read and enjoy it. Expect it before Monday.