This is part of the make your own guide to creating role playing systems.
Every game has a rule chain. From the simplest game of Pac Man to the most complicated version of AD&D, it's all built around this same concept. I've gone and refined it a bit to be somewhat clearer.
To make it really easy on you, I've broken it into steps. Basically, you follow these steps until you think your game world has enough complexity to keep your players interested.
This isn't about working out all the rules or the percentages. It's about having a guide that shows you how your game is played. Building a game with too much precision too early is a bad idea - that's how AD&D came into being, and that's why it's such a hodge-podge. This will help you lay the foundation for your game system.
When making a system, I suggest writing stuff down on a blank piece of paper "flowchart" style. I'll give instructions on how to do that, too.
1: The Primary Purpose of Conflicts
All games feature a central purpose - something that the players and GM try to accomplish. In D&D, it's reducing HP. In many more modern games, it's to tell a piece of story. There are thousands of possible purposes, choose one that suits you.
But remember, this is the primary conflict you're designing here, not the primary reason for playing. D&D's rules are built around reducing an enemy's HP, but the actual reason for playing is to have adventures. If "having adventures" is going to be the central conflict, then people would roll dice to have adventures rather than having adventures in which they roll dice.
I draw the purpose of a conflict in a triangle at the bottom of the page.
2: Basic Resolution
Now that you know the basic conflict, you need to know how it is usually resolved. For example, in D&D you roll dice to reduce HP, whereas in, say, FastLane, you bet chips on a roulette wheel for the right to tell a piece of the story. D&D's resolution is parallel - IE, enemies can reduce each other's HP simultaneously. FastLane, on the other hand, is perpendicular, in that if one person wins, another person (often the GM) loses.
This describes what kind of resolution systems you might choose, but anything will work, even rock paper scissors or making funny noises until someone laughs.
I draw resolution methods as circles above whatever they resolve, and connect them to their purposes with lines.
3: Add in a Modifier
A modifier is a method of giving a statistical benefit to one side or another. For example, a weapon in D&D allows players to reduce the HP of the enemy different amounts. In FastLane, having a chip pool allows you to bid various amounts of chips on the roulette wheel, allowing you to modulate your chances of victory and failure.
In D&D, rolling to hit is also a modifier. Instead of offering a sure but erratic bonus (2d6 vs 1d8 + 2 damage) it offers an erratic but sure bonus (you get to hit or not, period). This makes it more interesting than simply hitting, because now a beastie can have low HP but be hard to hit, whereas without this modifier all that mattered was the beastie's HP.
The basic rule is that a modifier lets a player express their intents and personalities more clearly. Having a chip pool lets a player be risky or safe, having a to-hit die allows players to specialize in damage or hits (or bypassing the need to roll hits).
I draw modifiers as boxes above whatever they modify, connected with a line.
4: Split the Approach
It's important to let players approach things in distinct ways by "going behind the back" of a modifier. This can be thought of as assigning modifiers to a modifier.
In D&D, the "weapons" modifier allows you to use different dice and bonuses to deal damage. However, having a weapon is not the only method of doing that. You can use techniques or be a mage. So, the "weapons" modifier has three "paths" - "weapons", "techniques", and "magic". Don't worry about the clumsy naming.
In FastLane, how you change the number of chips in your pool varies. The paths are "Betting/winning", "favor calling/granting", "Ability use", "statistic burn", "life risk", and so on. This is a lot of approaches, but you don't have to come up with thirty approaches at once. You can add more approaches at any time, and it's better to do it in small increments.
While a modifier allows a player to express themselves statistically, an approach allows players to express themselves categorically. They can choose to act like a monk, or a warrior, or a mage, or whatever. An approach can be semipermanent (such as choosing a class) but doesn't have to be.
I make approaches diamonds or rounded squares and put them near the things they modify, connected with a line.
5: Add in a New Conflict Type
Using any of the things you've created, add a new kind of goal it can resolve.
For example, in D&D you can use the "to hit" modifier to resolve noncombat challenges such as picking pockets or applying medicine. You can use the "weapon" modifier to do noncombat things such as healing, seeing in the dark, etc. Putting it under the "weapon" modifier rather than the "weapon" or "magic" approach means that any approach will have these noncombat methods, rather than only one of them. This allows for torches, spells of light, and light related techniques, for example.
In FastLane (or just about any game), the ability to improve your character by spending points is an end goal that gets added at this stage.
Be as specific or general as you like, and choose anything. Don't worry if the resolution method seems dodgy. That will all fall into place.
These give the players who choose different paths different capabilities, making them more unique. They are especially needed in places where you want the gameplay to be more complex than it already is.
I draw new conflicts as triangles beneath whatever controls them.
6: Clean Up
Look over your map. Does anything have an awful lot of connections to the same kind of shape? See if you can condense that into one or two much less specific shapes. An example of this would be the dozens of approaches to your chip pool in FastLane. they can be broken into two basic modifiers: adding chips to/from your pool and adding chips to your bet. Then those can have approaches.
Does anything seem awkward or lonely? See if you can move it to a more root branch. An example of this would be if you wrote "noncombat stuff" as a goal underneath "magic", and decided that it would make more sense if items could accomplish at least some noncombat stuff, such as nets, food, etc. So you roll the goal down to the "weapon modifier" rather than the "magic approach".
If something is misnamed, rename it. For example, the "weapon" approach and modifier allow you to address noncombat issues, so it's best to rename them "item".
This step is pretty important, as it will help you keep your game from getting too flabby and special-caseist.
7: Scrap it?
If what you see doesn't appeal or doesn't make sense, scrap it and start over. Seriously. It's no big loss.
8: Goto 2
Go back to step two and run through them again, but don't focus on whatever part you were focusing on before. Focus on whatever you want to have more depth.
For example, in D&D we can go back and focus on the "weapons" approach (now the "items" approach, since it's not all combat based any more). We don't want anyone to be able to get any weapon, so we add a resolution method: you can have weapons you buy or find. Then we can add modifiers to buying, so that we use gold pieces to try to buy stuff.
In fact, D&D is full of iteration. Range vs melee, saving throws, skills, stats - all of them are these same chains applied in slightly different ways.
This will also happen in game. For example, in the first tests of the first D&D, I'm sure there was no such thing as poisoned weapons. But then someone said, "I should be able to poison my weapon!" and hence poison was introduced. You can go in and add a poison modifier whereever it should be added, then run through the steps to make sure to balance it with approaches and limitations and so forth.
Here's a demo picture because everyone likes demo pictures.
Please notice that "favors" is an approach to the pool, but in the actual game, it's also fed by the pool. It's an approach and a goal simultaneously. To keep things clean, I just leave it as an approach. If you feel the urge, you can make the line a double-headed arrow.
This is mostly for YOUR use, so it only has to make sense to you.