SOTOPIA is an open-ended simulation environment and benchmark for evaluating the social intelligence of AI agents through complex, multi-turn social interactions. Introduced by Zhou et al. (2023), it provides 90 procedurally generated social scenarios spanning cooperative, competitive, and mixed-motive settings, scored across seven sociological dimensions.¹⁾)

Traditional agent benchmarks focus on task completion in isolation. SOTOPIA addresses a critical gap: measuring how well agents navigate the nuanced social dynamics that characterize real human interaction. Agents are placed in realistic social scenarios – negotiating, persuading, maintaining relationships – and evaluated holistically using SOTOPIA-EVAL.

Interactions are modeled as partially observable Markov decision processes (POMDPs), where each agent acts based on limited observations:

<latex>\pi(a_t | o_{1:t}, s_t, g)</latex>

where <latex>o_{1:t}</latex> are past observations, <latex>s_t</latex> is the agent's internal state, and <latex>g</latex> is the social goal.

SOTOPIA-EVAL scores agents across seven dimensions inspired by sociology, psychology, and economics:

The overall score is computed as:

<latex>S_{\text{overall}} = \frac{1}{7} \sum_{d \in D} S_d</latex>

Scenarios are procedurally generated with automated character creation, relationship assignment, and goal specification. This enables scalable simulation across diverse social contexts.

• GPT-4 agents underperform humans across most social dimensions
• Sotopia-RL training yields goal completion scores of 7.17 on the hard benchmark and 8.31 on the full dataset²⁾)
• Behavior cloning + self-reinforcement pushes 7B-parameter models near GPT-4 on goal completion
• Structured social context (S3AP tuples) improves performance by up to +18% on hard scenarios
• LLM-based evaluators approximate human judgment on goal completion but risk overestimation

# Evaluating an agent pair in SOTOPIA using the framework API
from sotopia.envs import ParallelSotopiaEnv
from sotopia.agents import LLMAgent
 
env = ParallelSotopiaEnv(scenario_id="negotiation_01")
agent1 = LLMAgent(model="gpt-4", persona="assertive_negotiator")
agent2 = LLMAgent(model="gpt-4", persona="cooperative_partner")
 
obs = env.reset()
done = False
while not done:
    actions = {
        "agent1": agent1.act(obs["agent1"]),
        "agent2": agent2.act(obs["agent2"]),
    }
    obs, rewards, done, info = env.step(actions)
 
# Retrieve 7-dimension evaluation scores
scores = env.evaluate()
for dim, score in scores.items():
    print(f"{dim}: {score:.2f}")

• Zhou et al. (2023) - SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
• SOTOPIA-RL: Training Social Agents via Reinforcement Learning
• SOTOPIA Documentation and Framework

• Generative Agents
• Agent Evaluation Methods
• Social Simulation with LLMs