I wrote a really long essay on mechanics inspired by my last essay, but it was too cumbersome to read. So let's talk about a Star Trek tabletop RPG, and I'll hit the same points with a clear example in mind.
Firstly, there's the concept of "class". Nearly every tabletop RPG has the concept of "class", because it's easy for the players to understand and the devs to balance. The issue is that some classes unlock new types of play: for example, only a magician can use spells, only a hacker can hack computers, etc.
Classes like "mage" and "cleric" are pretty seamlessly integrated into fantasy worlds because their abilities let them participate in the basic gameplay (fighting). A mage really sucks at fighting, but they are capable of throwing artillery fire into the mix.
On the other hand, classes like "hacker" are not integrated into Shadowrun very well at all, to the point of GMs just often just handwaving them as NPC-only classes. This is because the added gameplay - hacking, psychic stuff, and piloting - cannot easily integrate into the basic gameplay. A hacker sucks at fighting and has no easy way to contribute to a fight.
But hacker and pilot could actually be integrated into combat a lot more powerfully, and Shadowrun is slowly drifting that way. Use small combat drones, or perhaps live-tweak an allied cyborg to get every once of performance, or screw up enemy hardware, or create holographic illusions, or program heuristic nets in real time to predict and counter enemy actions.
Anyhow, our very first step is to make sure that all our "classes" can participate in our core gameplay.
... Our very zeroeth step is deciding what our core gameplay is.
Star Trek is not about combat. It's about seeing new things and meeting new people, and the conflicts that can arise. It's about finding common ground with aliens, no matter how odd: there is no "other" in Star Trek, no faceless masses to gun down. There are counterexamples, but those are usually from later on, after the series began to seriously decay. In the early days, even aggressive enemies like the Klingons were treated like people who didn't like you, rather than dangerous things you needed to kill.
As mentioned in the last essay, science fiction is about the life of people in the universe rather than about proving your own mettle. Therefore, rather than combat, we need the opposite: unification. Consensus. If D&D is like Pac Man, Star Trek is like Tetris. Rather than our basic action removing pieces from the board, our basic action needs to move pieces into alignment.
This also works well with the original combat of the series: it was u-boat combat. Submarines maneuvering, searching for each other in the dark. Pitched battles were rare and both sides usually came out pretty badly.
Unifying the pieces is a fun thing to consider, but we need to consider how that actually works. The advantage of a combat system as the core mechanic is that the GM can add and remove pieces pretty easily, without drowning in complexity. The plot of the D&D arc typically has very little to do with the combat, aside from maybe determining which pieces are involved.
But if we're unifying a large group of pieces, that means the GM is responsible for coming up with a lot of pieces, and setting them up strategically to offer a challenge.
We're not talking about a puzzle game with an optimal solution, but we're still talking about a game where pieces are related in a much wider variety of ways than "ally/enemy". The wider variety of options means that the GM needs to do a lot more planning.
Instead of thinking of each individual person as the same thing as a monster in D&D, instead consider specific "constellations" of people.
Adding individually would require something like choosing to add "Renault, the old-guard dignitary", then deciding who he is related to in what ways. Instead, we would have a "government in transition" constellation which would have several people stock - the old-guard dignitary being among them.
There are two keys to this approach.
The first is that a person can be in multiple constellations. Renault might be the old-guard dignitary from the "government in transition" constellation, but he could also be part of the "corrupt medical industry vs the plague" constellation, or "industry vs preservation" constellation. Which roles he plays in those will inextricably link the old guard to those topics, and add a lot of complexity to the situation.
The second key to the approach is that the GM doesn't have to preplan everything. The players don't meet everyone at once, and there's no reason to add them all into the game at once. The GM can use our "constellation" system to roughly plan out a scenario, then introduce characters as they are needed. The GM will always know what characters remain "in the wings", and can pluck out various pieces as needed to make the adventure more interesting.
This sounds a bit dry, perhaps, but it's the same as talking about how GMs might populate a dungeon with monsters. There will also be various setpieces and named characters, but it's a different topic.
The stats and classes the player chooses are important. Let's start with stats.
We want to choose stats which create the Star Trek universe's unique flavor. Basically, if people can be good or bad at something (a stat), then that thing is going to be very important to our setting.
"Piloting" wouldn't be a stat, because only a few people pilot at all. Similarly, "strength" is a bad stat because it almost never matters.
"Combat" would be a good stat. Spock and Data have very high combat stats because of their species. Kirk has a high combat stat because of his training and build. Bones has a very low combat stat. This variation in skill creates variation in results - not just winning and losing in combat, but whether combat is a route that can be considered. Even if there is no combat, the act of avoiding combat is gameplay in itself.
"Intellect" is a bad stat, because it's too vague.
"Expertise" and "Theory" are good stats, instead. Expertise covers your ability to work with complex technical systems such as repairing starships. Theory would cover your ability to do things in your head, like math, history, or tactics. Theory would probably cover medical expertise as well: Star Trek considers it almost a humanities job rather than an engineering job. So Bones and Crusher would both have a high theory stat, but a mediocre (Crusher) or abysmal (Bones) expertise stat.
We would need a variety of social stats, because a lot of the challenges will be social.
"Leadership" wouldn't be as limited as in most games, because you have a lot more allies than in other games. The engineering staff follows you. The ally you made by rearranging pieces counts. So leadership could be used to move allied pieces faster, further.
On the other hand, "Negotiation" would be how well you could affect pieces that aren't your allies (or aren't operating in unity at the moment), and visa-versa. So Kirk would have a high leadership, while Picard would have a high negotiation.
Lastly, "privilege" is a good stat for us, reflecting a combination of rank, self-assurance, and backing. People with higher privilege can act much more outlandishly and be forgiven more easily - it's our version of charisma. Even when in trouble, Kirk's high privilege meant he could continue acting like an ass and eventually he'd be back on top. On the other hand, Worf has a very low privilege, and therefore he gets stomped every time he steps even slightly out of his comfort zone.
Privilege is an unusual stat, but it helps to define the universe of Star Trek. It's especially important in NPCs, as much of the behavior of guest stars is dictated by their privilege. It gives the universe that slight "dystopian bureaucracy" aftertaste that it often carries.
We could use these six stats: combat, expertise, theory, leadership, negotiation, and privilege.
With those in mind, we can discuss classes and species.
For example, Vulcans and androids have significant bonuses to combat, expertise, and theory... but they have significant penalties to leadership, negotiation, and privilege. (Vulcans like to negotiate, but they're really not very good at it.)
Betazoids have empathic powers which, among other things, radically increases their negotiation and privilege - perhaps balanced by a loss of theory or expertise.
Klingons have a bonus to combat and leadership, but a penalty to theory and privilege. Their low privilege is their adherence to their own aggressive, weird culture - it doesn't mean they blend in. Their aggressiveness is a fish-out-of-water issue, not a high privilege.
Classes could start simply as the branch you choose. Command-branch characters would obviously focus mostly on leadership and negotiation, while engineering-branch characters would be mostly expertise and theory. Security-branch would be combat, leadership, and expertise, etc. Rather than granting bonuses to these stats, they would instead require those stats for their class abilities, much as a a magician has a high intelligence but being a magician does not increase your intelligence.
But... what's the grist between the stats and the conflict?
Every decent RPG has "grist". This is the stuff that lies between the stats you chose at the beginning of the game and the actual die-rolling you do while playing. In D&D, the grist is mostly inventory selection - getting the gear you want, choosing the spells you want, and so on.
Those things are enabled by your stats, and then you use those things in combat. Your direct action is to swing a sword. You have a sword because your strength is high and warriors get weapon feats. But you don't "roll strength" against the enemy, nor do you simply rub the number of weapon feats you start with on them. You choose a specific weapon and a specific feat, and apply them at specific times.
In our Star Trek game, we're the same way. Except that we don't care about inventory at all. Captain Kirk does not have a different phaser than Scotty.
Our grist is made up of interpersonal stuff.
Rather than choosing a sword or an ax, Captain Kirk chooses a threatening or gentle approach. Like choosing a sword, Kirk cannot easily change tactics - it'll screw up his rhythm and make the target think he's unreliable and weird. "Why's he acting friendly/unfriendly all the sudden?"
Rather than choosing four spells from a spellbook, Kirk will choose four techniques he excels at. Some might be based on his authority as captain: the threateningly low orbit, the photon torpedo spread, the surprise teleport. These are things he can use without "owning" the abilities, but the abilities make them fast and effective.
Some of his abilities will be based on Star Fleet authority: offering technical assistance, threatening with regulations, setting up trade agreements, having crimes dismissed, etc.
Some of his abilities will be personal: the two-handed chop, dressing everyone up in local clothes to blend in, seducing the locals, monologuing.
Everyone gets abilities. Some are species-related, like the Betazoid empathic power, the Vulcan mind-meld, or the Klingon berserker pain resistance. Some are job-related, like the time-dilation powers Scotty has or Bones' tendency to put people on sick leave. Some are personal, like Riker's charisma or Geordi's visor. Some are based on cultural or social authority, such as Picard's reputation or Guinan's knowledge of everything everywhere.
And a vast number of them are just personality traits like arrogance, loyalty, racism, etc.
All of these "abilities" are a bit different from abilities in D&D. Rather than being permanent additions to your character, they are more like equipment. The player will switch them out whenever it seems like a good idea.
Similarly, traits like "arrogance" are not just a flat role play guide. There are hundreds of things that could be called "arrogance", just like there are hundreds of variants on "short sword". It's possible to refine your arrogance infinitely, and eventually end up with the Star Trek version of a short sword +3 flaming +5 vs ogres. Something like "arrogance about job (+2) and ship (+3)".
Sometimes your abilities will fail you or vanish - just like a sword can critically miss or you can be disarmed. Some are more permanent, like Geordi's visor. Some are quite malleable, like whether Bones feels argumentative this week.
And, yes, sometimes you'll gear up with specific abilities when going on a mission. Kirk can decide not to bring his arrogance along this time, because it'd make the situation worse. He's still Kirk, he's just trying to be less self-absorbed today. If that doesn't work, then he can spend the next time he's alone for a few hours to reconsider his approach.
That's the key to enjoying this game. Your characters' personalities are not some monolithic obelisk. This is about interacting with other people, understanding them, and working with them. Like in real life, your personality isn't always raging out of control.
Mechanically speaking, the game works something like this:
The GM comes up with a scenario intended to last 1-3 sessions. The players participating place their 'business cards' (note cards with their character names) into the center of the table.
The players explore the situation by free role playing, and while that unfolds the GM places business cards for each NPC and unfolding situation they encounter. For example, if you reach a high-tech but non-spacefaring planet where there is a disease raging out of control, the GM will place the disease as a card, as well as the various individuals the players meet, such as the planetary governor, the lead medical researcher, and the man dying of the disease. As time passes, the GM may add the military commander, robot police forces, and an oncoming hurricane.
Cards that are neighbors long-edge to long-edge are tightly allied (or being watched) and within shouting distance. Cards that are aligned on their short edge are more loosely allied. Note that tight allies chain - a giant stack of tight allies are all allies with each other. But loose allies do not: you can never be loosely allied with more than two people.
This is a player-focused arrangement, so if there is hidden information that only the GM knows, it doesn't have to be reflected in the cards until it is revealed. Of course, since it's player-focused, much of their efforts will be to align these cards to suit their needs.
Sometimes this is easy: if two players want to team up, they just agree to do so. Assuming no NPC or situation interferes, it's just a simple bit of free role play and the players' cards butt up.
Similarly, moving your card out of alignment is easy - going off on your own, breaking an alliance. It's just some free role play and you can move your card. But your erstwhile allies might not take it well, depending on the situation.
The real challenge is when you are trying to align yourself with non-player cards.
Bones wants to check out the disease, so he decides to move himself vertically with the disease. This doesn't mean he's allied with the disease, it just means that he's going to be in the midst of it. To do this, he'll need to explain how he's finding infected people.
One approach would be to talk to the lead medical researcher. Another would be to talk to the man dying of the disease. If they are okay with it in free role play, then Bones can be led to the disease and hook up with it without ever needing to leave free role play.
But there is plenty of strife. The dying man doesn't want to talk, and the researcher says it's too dangerous to expose him to. Now what?
One option is elbow grease. Bones can strike out into the city (perhaps with some friends) to search it manually. This is a shift action (~6 hours), so Bones expects it to take a while.
If Bones can use an ability to help out, this stands a pretty good chance of working. If he has a specific kind of medical scanner ability, that'd be fine. He could also use a less specific ability - such as medical intuition or intent to help everyone. The downside of the broader abilities is that he must have them equipped to use them, which means they can be used against him by the GM later on. It's much harder to use a medical scanner against you.
Bones has to roll against a particular target number on a D10. This number can be higher than 10 or lower than 1, but you may still want to roll, because the amount of failure or success matters.
Each point of success or failure can be spent to reduce the time spent or to increase the result's effect. If Bones succeeds by only a little, it could take him 6 hours! But if he succeeds by a lot, it might take him only a few minutes, or maybe he discovers a massive number of sick folk.
If Bones fails, he'll get in trouble. There's no such thing as "merely failing". If he fails by a small amount, he'll get arrested after 6 hours. But if he fails by a large amount, he might get shot after only a few minutes! Bones chooses how to spend his failure points, and he has to decide whether to fail fast and harmless, or slow and painful. Failing slow is really useful, because that means someone can interrupt his search before he actually gets caught, essentially negating the result at the last second. This requires that person spend "convenience points", but it's better than getting shot.
Anyway, that's if Bones has a suitable ability. If he doesn't have any suitable abilities, he can fall back on pure stats - probably privilege or negotiation. This may or may not succeed, but either way you ALSO fail by the exact same margin. If you roll a 3-margin success, you also have a 3-margin failure. If you roll a 3-margin failure, it's just a failure. This is the punishment for over-reaching.
Let's say Bones doesn't have any suitable abilities, but would prefer not to risk the automatic failure. So he instead chooses to win over one of the people who might be able to help him.
Let's say he chooses the lead researcher to buddy up with. He could also tail them, threaten them, etc, but let's say we're going the friendly route. The goal is to form a loose alliance (short edges of their cards butting up). The lead researcher is pretty antagonistic at the moment, so this might not be easy. How to accomplish it?
"Form a loose alliance" is a specific action, and there is a specific set of approaches to doing it. The negotiation stat is usually the backbone off this approach. Each time step dedicated to it reduces the difficulty - if Bones negotiates for a scene, he gets +0. An hour, +1. A shift, +2. A week, +3. In addition, he'll want to use an ability, or he'll run into the "autofail" problem again.
Well, Bones doesn't have any real negotiation abilities, and his negotiation stat is rock-bottom. So he has to fall back on privilege, which can always be substituted into these kinds of situations if you don't mind a small penalty. By ignoring cultural and situational barriers, Bones can just hang out with this researcher and try to make friends. Bones has a very high privilege - he's a grumpy old luddite weirdo that everyone loves.
Bones decides to hang out with the researcher for a phase to ally with him.
If this seems too hard, Bones can soften the researcher up by de-escalating. Escalating and de-escalating are important tactics to control the situation without directly moving cards. De-escalation is done by focusing on shared elements and ignoring potential thorns. Both Bones and the researcher are medical, and that means they share a job that relies on theory. Bones can use theory (and, hopefully, some ability) to de-escalate, then use privilege to form a loose alliance.
Even if Bones has no abilities to help with this, he can juggle timing to make it work out. He gets smacked with an autofail if he works without abilities, but that's okay. He'll spend all his success points on lowering the time to success, then spend all his fail points on increasing the penalty for failing. Now he has a few hours between his success happening and his failure happening, and in that space he can take advantage of the alliance.
Well, anyway, that's just a very basic set of ideas. I haven't polished it much.