I love mods, and people are talking a lot about mods, so let's do more talking about mods.
A lot of people are annoyed that Unity and Unreal don't make modding easy... but modding has never been "easy".
Flash, CryEngine, IdTech, RPGMaker... none of them are "moddable" right from the ground up. Modding has always been something the devs decide to include.
With that in mind, we can talk about mods in two ways. From the perspective of the person installing the mod ("player"), and from the perspective of the person making the mod ("creator").
---
From the player's perspective, there are probably three kinds of mods:
1) Piecemeal Content
This is content that will almost never conflict with other piecemeal content, except in the most trivial ways. Piecemeal content is probably the most popular kind of mod, including things like recruiting other players' characters, sharing messages about sun-praising, downloading vehicles from the workshop, adding new skins, etc.
2) Local Content
This is content which will conflict with any other content in the same locale. For example, if you install a mod to turn you into Bayonetta and a mod to turn you into Link, they won't get along. Similarly, you can't be in two levels at once, two missions at once, etc.
Local content can be turned into piecemeal content if you add a tool to help the player manage this kind of conflict - for example, Bayonetta and Link can both be "skin options" instead of simply overriding your appearance.
3) Core/Process
Some mods change fundamental rules of the game. For example, Skyrim mods that make character progression different, or change how weather works, or make you need to eat and sleep, or make the shaders work better.
Sometimes the mods are completely invisible, such as mods which allow other mods to work, or mods that clean up memory usage.
Frequently, core/process mods are part of a chain of required mods, like 'install the event mod, then the silent talking mod, then the progression revamped mod, and only then you can install the custom skill pack'.
---
The reason to think of mods like this is simple: it helps us to think about how mods work within our game. Can our game support piecemeal content additions? If the gameplay allows for certain kinds of piecemeal content, what is required to surface that and allow the players to load it in? If the game can support local content, can we change the way our game presents it to turn that into piecemeal content?
Can our gameplay support core/process mods? Can our game architecture? Can we revamp it? Can we create a tagging system so that we can have mods say what other mods they require, at what version?
The answer is rarely a flat yes. This kind of thing is a bit difficult to engineer, and it may damage your core gameplay. But even a small amount of moddability is a good thing, and you can leverage it to either make your game more appealing or make your game stickier.
For example, in Guacamelee, the only kind of moddable content is custom skins. However, they sent out custom skins to popular YouTubers, enticing them to play the game and be more positive about the experience. The modding made for great outreach.
---
The other half of the equation is how the creator of the mod thinks about the mod. This is equally important.
1) Diegetic assembly
When the creator never leaves the game to create the content. In the best case, the content is created simply by playing the game, but nearly all of the time this is an in-game editor. For example, you create your character, build your ship, assemble your house - all using an in-game editor.
The key here is that diegetic assembly happens in the course of normal play. A character builder counts because every player will use it at least once and consider it part of their playthrough, their experience. A mission editor generally doesn't count, because it's never used in the course of playing the game.
There is a challenge to keep the editor simple enough for everyone to use but powerful enough to let players create complex or nuanced results. There is also a challenge to leverage the editor: a character editor for a game where you only create one character is not as well-leveraged as a game where you create multiple characters over the course of the game.
Compare The Sims' character editing to Fallout 4's.
2) Tool-assisted assembly
The creator uses a separate tool to assemble the content. This tool may be in-game, such as a mission editor. It may be out-of-game, such as photoshop or visual studio or even just notepad. Either way, this is a thing the player has to go and do, separate from playing the game.
This is quite a hurdle. Few players will go and use a tool that is not required over the course of play. Because of this, it's usually ideal if you can leverage a tool in both the main gameplay and as an asset creator. Normally this is done by using the same tool, but having a creative mode where the player's specific play constraints are relaxed.
Done well enough, this turns into diegetic assembly.
Core/process mods are a good example of mods that are almost always created with tool assists. It is exceedingly rare for a game to allow its own rules and progressions to be overwritten in the course of play. Normally these mods are created with visual studio or notepad, the code then injected back into the game.
3) Hacked assembly
If the creator has no way to add their mod to the game, they may hack the game to force it to accept the new content.
You might consider this similar to a tool-assisted mod at first glance, but a tool-assisted assembly uses an injection method the developers intended for modders. That text you edited, that DLL you compiled, the game goes out to look in that directory for those things because the dev decided to support those kinds of mods in that way.
Hacked mods usually take advantage of things the dev accidentally left available instead of intended for mods. For example, they might be "trainers", hacking the game's memory space to give you infinite health or cash. Or they might overwrite the game's core assets with new assets.
As a game dev, hacked assembly is something to avoid. Mods assembled via hacking almost always conflict with other mods just due to how they work, so it's best to try and create a proper interface for mod injection.
---
When I think about mods, that is how I think.
What kinds of content can be injected? What kinds of tools can I use to make that injection easier, more useful, more potent?
What methods of creating content are there? What kinds of interfaces can I create to help modders create mod packs and chains of mods that don't conflict?
Unfortunately, no game engine natively supports this kind of thinking, because it's part of every individual game's unique design. There's not really any low-hanging fruit here: you have to design your game to be modded.
