How do you make social NPCs? Here's a new approach.
Fundamentally, social is a matter of conversation. It's not back-and-forth: it's simultaneously offering a bit of your mind and looking to see how the other person is receiving it. You may simply offer bits of your mind over and over and over, with no "balance" of them offering pieces of their mind in exchange. All they are doing from the perspective of an outside is listening, but from the perspective of the talker they are offering feedback. They are offering their judgment and acceptance and incredulity and so on, all in line with the conversation.
So, to simulate social NPCs in a game, isn't it necessary to allow for this kind of conversation? The kind of conversation where you respond not by taking control of the conversation or having your line-item dialog tree option, but instead allowing the player and the NPC to understand each other's intentions via an ongoing passive response.
Here's an example. An friend comes up to you and says "My front window broke last night!" As this is being said, the player enters their response. It comes out as "oh?" or "whoa!" or "huh!" or "so?" or whatever. Then the friend goes on... but what they go on with depends on whether they think you care, what they think you want to hear. So they may go on with "... so, that was odd, yeah. How's life?" or they may go on with "I woke up and there was glass everywhere!" or they may go with "Maybe someone is throwing bricks at my house?!" It all depends on whether you seem to want to hear more.
There's a lot of stuff you need to do in order to build this into a game. You need to give it both clarity and texture. So the player understands that answering "oh?" and "whoa!" are going to have two different results, or at least result in two different perceptions of the player by the NPC. Moreover, the actual conversation needs to be flexible enough to adjust to the emotional or personal interest given. This can't realistically be done with a dialog tree - the explosion of options makes it impossible, especially if you use the rule where if you hit the same interjection button multiple times it makes you give the interjection more emphatically. Having four different options per line is bad enough, but having 8 or 16 would be impossible.
You can do it by writing several chains of dialog, though. For example, you might write a technical chain and an emotional chain. Each link on the chain is heavier than the last.
So if the NPC says "My front window broke last night!" you might indicate technical interest: "oh?" or you might indicate personal interest: "are you okay?" This would determine whether the next line is taken from the technical or emotional chain.
Moreover, if your interjections are more emphatic, it simply means to skip some of the links. The NPC feels comfortable presenting a heavier link because she knows you are interested enough that you won't consider it a burden or boring. So if you hit the technical interest button a bunch of times, you might interject "wow, that must have been a hell of a mess!" And rather than simply taking the next link on the technical chain ("there was glass everywhere"), she'd respond with a link a distance down the line, something like "well, cleaning it up was a sure annoying. That stuff can cut you!"
When you are feeding one chain, pulling down links for conversation, the other chain remains still. So if you express technical interest for a while, but then say "are you okay?" they won't skip to the 20th emotional link, they'll just move to the first one. However, you can stay on the technical chain by expressing more technical interest but still expressing some personal interest, which might end up pulling the emotional chain along without actually causing any of the links to be brought up in conversation. Or visa-versa.
The key to this is that both technical and personal chains have in-game reasons to pick them. A link that is brought up in conversation has an in-game effect. A technical link would improve the NPC's ability to resolve the situation faster and perhaps grant them skill experience or emotional stability. An emotional link would deepen your relationship with the NPC, but cause them to get emotional, upsetting their efficiency at in-game tasks or losing some skill experience or extending the duration of the situation.
The conversation isn't unlimited, either. Your relationship or their energy level or whatever determines the maximum number of lines they'll say. So you may want to leap ahead to the heavy links as soon as possible.
I would actually make the links visible on the screen for the player to see. That way you can put in broken or damaged links. Conversational bits that will abruptly end the conversation or cause offense or something. You need to avoid having them brought up, so you need to skip over them or spool past them while on the other line.
In this way, the conversation becomes an actual game, where you are talking to someone and it has flexible in-game results. It's not just a matter of +10% bargaining or whatever.
More complexly, this system could be built to allow the player to talk and the NPC to interject. This would be a matter where the NPC can see the broken/damaged links in the conversation and their interjections are partly based on personal preference and partly to give the player hints that there might be damage on a given chain in a certain number of links. So the player chooses which chains to advance how many notches, taking into consideration which links he thinks are broken and how much the NPC gives a shit.
It could also be adapted to allow for back-and-forth conversations, although this would require an actual conversation generation system.