Generally, the first thing I think of when they think of a role playing system is how the challenges are resolved. D&D is just about synonymous with a d20, for example. Other games use Nd10, rock-paper-scissors, decks of cards - just about any method you can think of.
In choosing your own challenge resolution system, you'll need to decide what kind of feel you want the game to have. Here are some common fragments you can consider when designing your own fundamental resolution system:
Flat Opposition. A lot of games use flat opposition which must be exceeded, especially on non-combat situations. Flat opposition is when the quantity you need to exceed is known. For example, "you need a 17 to hit" or "you need 4 dice of at least 5 to hack the mainframe". This is useful when you're playing a beer-and-pretzels game where the players don't want to think real hard.
Decaying Flat. This is the same as flat opposition except that if you don't reach it, the opposition drops some. Hit points are an obvious example of this: you need to do 150 HP of damage, and if you do 75 for two rounds, it counts. Obviously, you can mess with these numbers - include regeneration, thresholds, maximum damage levels, powers that unlock at high levels of damage... This is a bit more number-crunchy for the GM, so this is generally kept to a minimum in most games.
Opposed Simple. An opposed simple is when all the sides contribute numbers (usually die rolls) and add their bonuses to them. The highest wins. This has a much larger element of chance than a flat opposition, which is usually best to reign in by minimizing the randomness - making the die roll roughly equivalent to the highest bonus, or letting people play a card out of their hand.
Recursive Flat. This is when the success of one side's flat opposed roll becomes the flat target for the next side. For example, "Dave got 5 successes, so in order to defend himself, Jed has to get 5 successes, too." It contains less randomness than opposed simple, so it is suitable to dice of any size, but it can make turns take a long time. It's good for games where you want the players to always feel that they have the power to change things, no matter how late in the game things get.
Recursive Opposed. This is when players repeatedly bid. For example, each player can play any number of cards from his hand, then his opponent can, and so forth. This can be done with hidden data (cards face down) or with revealed data (die rolls/cards up/tokens). However, if it is with revealed data, there should be multiple simultaneous challenges or long-term bid conservation to prevent everyone from bidding until exhausted.
Single Die. Generally a d10 or a d20, the single-die method is fast and easy, and usually combined with flat or decaying flat opposition because of its high randomness. Also, most games allow for the maximum and minimum rolls to be considered critical, to punch up the variability. Generally, single-die methods do NOT allow for exploding dice. This is most useful for beer & pretzel games.
Multi-Die. Can be any die, but usually d10s or d6s. How many dice you roll is usually related to your skill level. Multi-die specifically refers to dice where each die can individually succeed or fail, rather than summed dice. The number of successes (and often failures) is usually a part of the final resolution. Generally, if the type of the die is d10 or higher, it uses critical successes/failures, whereas if the type of the die is a d6, it uses exploding die (ie, every max roll allows for another roll that adds on). This is simply because of the frequency of exploding dice. These are usually used by moderate games where the players are fine with thinking, but not too hard.
Variable Dice. Many games use dice that vary based on the character's capabilities. For example: damage is a d6+1, or 3d8+9, or whatever. Unlike multi-die, they are added together rather than independently counted. There were a few old systems which used a variable die method that was always the same (IE 3d6 for everything), but this has gone out of vogue. This is most useful for a system where you want a huge difference between people who are good at something and people who are bad at something.
Uneven Dice. Uneven dice is a type of multi-die where dice rolled have different rules applied to them. For example, you might roll two white dice and a blue die, with the blue die meaning something specific. Or you might roll 2d10 and 3d6... this kind of system can produce very interesting side effects, such as allowing players to choose which dice to roll.
Random Card. Using a deck of playing cards allows players to get a very different kind of randomization. Playing cards are easy to assign unusual values to - for example, "the joker is a botch" or "the king is +10 and draw another card" or "hearts restore a hit point". While theoretically cards can be counted, in practice this is too difficult for 99% of players. Cards can be either played face up in turns, or face down simultaneously. These systems are useful for games where you want lady luck to play a more entertaining part than simple criticals and botches.
Hand of Cards. Similar to above, except that players are given a hand of cards to play from. This allows players to spend or save their "good" "rolls" as they choose, and adds a whole new element to the game. In addition, if players are allowed to trade cards and see each other's hands, they start using dramatically more team work and will have a better chance of estimating which cards remain in the deck. Very much a thinking player's method.
Multi-Card. Some games allow players to play multiple cards and gives the players bonuses for the relationships between the cards. IE, poker hands are worth more. This does slow the game down, but allows players to control their gameplay exactly. Also, it allows specific characters to have specific abilities: IE, "the sword of Bronenogry deals double damage if all cards played are hearts." This is the ultimate thinking player's method.
Tokens. One of the fastest ways to play if you're playing a more iterative model (IE, each player can respond to what other players do) is to use tokens, with each character have a certain number of specific kinds of tokens - replenishing a certain amount each round. Tokens can represent "a point", but they more often represent an unknown number of points to be resolved at the end of round. Typically either a D6 or a 50% chance of being a point (red/black cards). This does well for minmaxers and wargame fanatics.
One of the basics I didn't mention but you might have picked up on is how much control a player has over what they put forth. For example, if all players roll 1d20 to hit, then they have relatively little control outside of trying to get their character stats to have bigger bonuses.
On the other hand, if they can play one of the cards from their hand, they can choose how "lucky" they want to be at any given time, using up "luck" as they proceed. Because this is a known quantity, these should always be actively opposed so that the player cannot flatly calculate the minimum play required for success.
And at the top of the player control system is tokens, where a player will be able to bid specific amounts of his resources on offense, defense, movement, damage, whatever. This allows a player to specifically determine exactly how his character should play in excruciating detail - something that some players love and some players hate.
There are other methods I didn't cover here, such as rock-paper-scissors. These kinds of methods are generally more suited to LARPs and are usually a simplified version of one of the above or a basic guessing game of some variety.
Also notice that I didn't cover things like whether you're using miniatures and maps or not, or what kinds of challenges you allow, etc, etc. Those are important considerations, but this is unrelated.
Choose a method that suits you and your players.