I've been thinking about gameplay complexity. It's a big question for me, because most of my games are not strict genre games.
Genres acquire a lot of complexity - players become familiar with the standard play and that lets you add more complexity on top of it. A good example of this is any recent Street Fighter game: cancels, partial supers, half-dozen bars that go up and down arbitrarily, gem power-ups, counterthrows, tag juggles, dizzy mechanics...
Street Fighter is a good example of the issues involved, because fighting games are something everything thinks should be easy to understand... then they try it out and they don't even realize that a "throw" is a thing, let alone a partial-super-rolled-into-a-cancel-quick-throw-followed-with-a-tag-juggle-into-a-full-super-filling-a-gem-quota...
When you are building a non-genre game, everyone is that person who thinks it should be easy to understand. There are no people that already know the rules.
You can design a simple game. It is possible to make a compelling, simple game. Threes and Triple Town both use very simple, approachable mechanics. These are great designs, but they are simple games. They use randomness and tight constraints in a very tight loop.
Another reason to go simple is to have a smooth curve, and introduce more complexity as the player gets used to it as it is. This leads to half your game being a tutorial, but more than that, it's not a very good way to do things.
In every game, you'll hopefully be exploring a particular kind of experience or play. If your game is a genre game, you can build off of genre play to explore something at the fringes. A good example of this is the proliferation of 'shtick platformers' where you have all the normal platforming play plus one trick.
But in a non-genre game, there is no base foundation of "platforming". Whatever your core experience is, it's going to be what you're exploring. If you try to explore that and then make it more complex, that's not a good formula. You're trying to create a genre in the first part of your game, and that's not going to work out. Not least because your player simply can't internalize a genre that fast.
That isn't to say all non-genre games have to be simple!
There's a lot of power in complexity. However, you need to be careful when you approach it. If you want a non-genre game that has progression and is kinda biggish, there are two approaches you'll probably think of, three you might not, and a bad one.
1) Width. This is when you add more kinds of the same play. For example, in an RPG you add enemies with different stats, you have several different modes of attacking, you have several different numbers that need to be optimized, etc. In TripleTown, you have many different kinds of combinable resources.
2) Constraints. This is when you vary the constraints to pace the player and guide them through the experience. In an RPG you move from town to town, each one with different enemies, different equipment, different visuals, even different party members. The most common constraint is randomness: Threes and TripleTown both use randomness. RPGs typically have random battles.
3) Emergence. Emergence is a bit complicated because it's very easy to think you have emergence when you don't. RPGs almost never use emergence because they want a tight grip on the pacing and progression, and it's hard to predict exactly how things will emerge for each player. Still, it's valuable: Threes uses emergence because the player's previous accomplishments leave ever more high-number tiles clogging the board.
4) Construction. When the player creates something, it creates an effect on how the other parts of the game play. Examples include leveling up a character, building a rocket, forming a party in WoW, etc. This can be tightly or loosely controlled, so it is important to know how much freedom to change the world you should give the player.
5) Multiplayer. Allowing players to compare themselves to other players or incorporate other players' choices can be very interesting. Sometimes synchronous (perhaps even local) multiplayer is the answer, but don't overlook asynchronous or implicit multiplayer if your game involves creating content. Dragon's Dogma does this reasonably well.
6) Maze of Actions. A lot of games aim for complexity by simply being incredibly complex. An example of this would be Street Fighter, which has a massive number of different elements in play all the time, making it almost impenetrable for a new player. None of the kinds of play are variations on a theme - each is a completely different thing to consider - gems, counting events, canceling, supers, block-breaking, throws, tag teams... they do combine into one experience, but it's convoluted as hell to understand all the pieces that flow together.
In general, using several of these is a good idea, as each is more or less effective for any given player.
The reason I'm talking about this in such detail is because I'm coming at it from both sides today.
I read an article about "simplifying" RPGs - removing most of the numbers from them. And I'm also considering how to build my xenodiplomacy game, which isn't a genre title.
When it comes to simplifying an RPG, you need to consider all the complexity that RPGs have built up over the years. Most of an RPG's complexity comes from width and constraints. Balancing stats is one of the core play elements, although it's made interesting less by being hard and more by being wide: there are dozens of stats and hundreds of equipment options that affect various stats in various ways.
If you were to remove that or dumb it down, you would be removing one of the core complex aspects. This would make the RPG easier to approach, but it would also reduce the long-term complexity of the game. To deal with that, we would need to add meat somewhere else in the title. Perhaps widen another kind of play (the interpersonal conversations?), introduce some emergence/deeper construction, or add in some multiplayer elements.
This is far from being a theoretical problem. Dragon's Dogma did exactly this. It radically simplified stats and talent suites, and added in a mild asynchronous multiplayer element in that half your party was made up of characters other players had designed. Dragon's Dogma also widened the combat system, having three kinds of intertwined combat (melee, ranged, and anti-titan) plus support rather than just one plus support.
Well, what I'm trying to say is that removing complexity from a genre title makes it more approachable, but you need to be careful. The balance in a genre game is very carefully grown from decades of experience, and hacking off bits willy-nilly results in a really awkward result.
On the other side of the spectrum is non-genre games trying to add complexity.
I have a xenodiplomacy game brewing in my mind, but it's difficult to have the right amount of complexity. Too little complexity and the game grows stale. Too much complexity, nobody can figure out how to play it. This is made more complex by my insistence on having social characters - that is, characters you can get to know, can affect, and that can affect you.
As normally considered, socializing is "non-core". That is, it's not really gameplay: it's a pacing system. As the player progresses through the core game, they steadily have more opportunities to socialize with their allies. But in order for that to work, the core play needs to reflect those characters and reflect onto them, so that they matter. See: every Ubisoft RPG.
In general, this is handled very badly. The concept of an "open party RPG" is deeply flawed, because I don't know anyone that uses a majority of the characters: everyone settles on their favorite party and that's that. This means that all the other party members are almost completely detached from the core gameplay. Same problem as a dating game: you're supposed to ignore 90% of the characters.
Just off the top of my head, I can think of ways around this. Have only 3 NPCs. Force the player to cycle NPCs. Allow a tactical role for every NPC. Etc.
The other solution is to do the opposite. Make the socializing core.
This is an interesting challenge because it's not really the approved approach. Even in dating games, it's not usually about socializing. It's about grinding for stats or cash or whatever. The socializing is just canned dialog -> A, B, or C choice -> stat/item check.
Creating social play is a really interesting challenge, and in a xenodiplomacy game it might actually make a fair amount of sense. It's not about simulating a conversation. It's about two aliens trying to make each other more comfortable, even though they have almost no method of communicating directly.
Social play doesn't have to be about jabbering face to face. It can involve base-building (a comfortable habitat), body language, creating customized gifts (not buying canned gifts), planning safe and interesting events... Or, of course, it can be some kind of simulation of painstakingly mistranslated conversation, whatever.
Whichever way you take it, though, the social play is non-genre. This means that if it's very complex, the players will feel extremely lost. So if I do make social play more than "choose A, B, or C", I have to take a very careful path towards complexity. Width, constraints, emergence, construction, and multiplayer are all on the table, but it's up to me to figure out what the hell to actually DO.
Even if I back off and make it non-core, I still need core gameplay that ties into them. That's going to be complex, so it might be best to steal a genre for that part so the players don't feel lost.
Well, my instinct is that the social play is the same as the diplomatic play, just with a different set of constraints. But that's as far as I've really gotten so far.