StreetSmarts, a GTA mod inspired by SentientStreets

Character output is generated by LLaMA 3 8B Instruct running on an M1 Mac. The code was quickly hacked together.

Recurring characters are scattered throughout the map with saved chats, which means you can continue conversations with them and build relationships. Conversations with non-recurring characters (like random pedestrians) aren't saved.

For those unfamiliar with LLMs, model outputs can be tweaked with prompts. Here the model can be guided to output only in JSON, only speak in pig latin, or in this case, roleplay as a character within Los Santos.

NPCs can also take actions based on the state of the conversation. Currently there's 3 actions: fight, follow, and run away. Some characters run away when threatened, and others are easily angered and resort to fighting. More outgoing characters are likely to accept an invitation somewhere, while others aren't so interested.

The simplest way to calculate character actions is to feed outputs back into the model to output probabilities for fear, anger, follow. The output looks something like this:

{"angry": 0.92, "scared": 0.15}

The results were fairly accurate; if I were to guess, around 90% of the actions taken (attacking or running away from the player) felt normal and not out of the ordinary.

I stopped working on the mod, but there's still more to do:

• Voice input
• Text-to-speech output
• Use a better model, like Gemini Flash, which is faster and has vision capabilities
• More actions (cheer, dance, shoot, lead, etc)
• Custom missions like SentientStreets

Video games are very obviously going to leverage LLMs soon. Unfortunately GTA 6 is coming out 2 years too too early -- a worthy successor to GTA V deserves to have better NPCs.