The reason I keep building tabletop RPGs that nobody really plays or sees is because tabletop games have an interesting clarity. A lot of the most time-consuming stuff in a video game (or even a published RPG) is not required when you're building these kinds of prototypes, so you can explore a lot and polish a lot.
The big thing I keep thinking about is the difference between elegance and texture.
One of the impulses I always have is to aim for elegance. I always want a set of simple rules that work in all situations and create a lot of tension and emergent complexity. But that's not necessarily the best thing to do. It's very bland.
It's true that these rules make the game pretty easy to pick up, and it's also true that the game can be very interesting if the players and the GM inject themselves into the world. But it's hard to get traction with a system that tastes like tofu.
The other end of the spectrum is oldschool AD&D, where nearly every rule is special-case. The mechanics governing armor class are completely unrelated to the skill check mechanics which are completely unrelated to the class restrictions which are unrelated to the loot drops and so on. It's not just a stapled-together set of independent mechanics, it's also a bunch of special cases within those mechanics.
To a lesser extent, any game with a specific world is full of this kind of special-case stuff. "Oh, this particular city specializes in this particular thing, and has this particular relationship with that city..." "Oh, the prestige class 'bone knight' has these abilities and is widely considered by the public to be..." "These forests are full of lurk-spiders, which are giant spiders that also have a chameleon ability..."
A lot of people might consider this to be "color", but it is often very integral to a player's experience.
First, they're important narratively. These things will often be what the GM uses in her campaigns, and will drive small or large plot elements. Even if they aren't used directly, the GM's own scripts will tend to reflect the examples provided.
Second, they're important statistically. A lot of these special-case rules flavor the choices the players make. "I need to gain two points of strength so I can use this particular skill", for example. Or "we better stock up on jorts, they're heat resistant and we're headed into a jungle." Or "I'm learning this worthless cantrip spell because I can use it in these unexpected ways..."
Third, special cases are important to the "stewing" part of the game.
"Stewing" is a concept I've not really seen explored anywhere. Basically, when someone gets interested in a system, they don't simply stop thinking about things the instant the session ends. They keep thinking about it. They think about it a lot on their own time. They imagine their path forward, alternate builds, imagine potential scenarios, read about interesting monsters and places, maybe read the novelizations if they get really interested.
This isn't about the GM actively scripting a session. It's about just stewing in the system.
In my opinion, stewing is the purest example of traction. The more rewarding it is to stew over a game, the better the traction is. I think this same traction shows itself in gameplay and session scripting too, but its purest form is when a player borrows the rulebook so she can read it on her own time.
As far as I can tell, in order to improve traction you need to have a particular kind of world design, a particular kind of special case.
Basically, it's the special case that makes people think of a scenario.
A special case rule might be a critical hit table full of generic hits. Critical to the leg, -1 dex; to the arm, -1 str; etc. But this is pretty flavorless: nobody reads that and thinks about a scenario.
A better table would be one like the Hackmaster tables, which are deeply hilarious, pages-long tables that include hit locations like "left nipple". But even these are probably not ideal.
If you want the best critical hit table, you need hit locations that make the player imagine what it'd be like to get hit with one, or hit a monster with one. "Leg, -1 dex" isn't going to make anyone give a shit. But "knocked down and flung 10 feet backwards" would make them think.
It's important not to drown them in options. There's a reason monster descriptions tend to be most of a page and include a lot of extraneous details: if it's just an endless list of names and stats, the player will lose focus and skim. It's the same with tables.
You don't need a "flung 5 feet backwards" and "10 feet" and "20 feet" variations. You just need one option that punches that particular spot. The next option does something else, such as "concussion: cannot use skills for 5 rounds".
To be honest, a random table is also not a very interesting way to handle critical hits. Instead, consider linking critical hits to the weapon. Each category of weapon has its own critical hit. The hammer class knocks people down, the staff class concusses them, and so on.
By linking the events to other in-world choices, you accomplish two things. First, you make those choices matter more. Second, you make the assumptions of the player matter more. For example, many players will just assume a particular weapon for a particular character. Now they have a piece of rules meat attached to that assumption, and they can easily imagine the results. "Oh, dwarven warrior with a hammer... cool, he'll be blasting things away from him!"
The thing you have to watch out for is taking it too far. If everything is special-case rules, then you can only play if you've already played before. No new players will really be able to get a feel for the system.
Five ways around this:
First, have a really easy to understand set of core rules. The texture comes out during play, not during initial explanation.
Second, have templates ready so a player can choose what they want without understanding all the rules. Make the templates decent.
Third, make the special case rules imply scenarios. A rule like "+3 to attack" means a lot less than "nightvision". Nightvision implies a lot of scenarios. +3 to attack is a statistical tweak. Even if +3 to attack is stronger, it's not as vibrant.
Fourth, there's nothing wrong with statistical meat: just hide it far along the player's career. Everything a new player is likely to encounter should be vibrant and scenario-inducing. Later on, when they deeply understand the balance of the system, +3 attack will make them drool and think up cool scenarios just as much.
Fifth, add in a hint of the scenario if you don't think the special case gives enough impulse on its own. Old Shadowrun books used to do this a lot - the endless list of guns or districts or whatever would roll by. But at the end of each entry would be some chummer snarking over the net, gently giving hooks. Like "GUNCHILD_88>> Oh yeah this was the place where the cops did that mafia sting... never been the same since!"
Anyway, those are my thoughts.