SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems
Provides a benchmark for evaluating social reasoning in embodied multi-agent LLM agents, revealing critical limitations in deception detection and planning.
SocialGrid benchmarks LLM agents in an embodied multi-agent environment inspired by Among Us, finding that even the strongest model achieves below 60% accuracy in task completion and planning, and agents fail to detect deception at near-random chance. Planning assistance improves task completion but social reasoning remains a bottleneck.
As Large Language Models (LLMs) transition from text processors to autonomous agents, evaluating their social reasoning in embodied multi-agent settings becomes critical. We introduce SocialGrid, an embodied multi-agent environment inspired by Among Us that evaluates LLM agents on planning, task execution, and social reasoning. Our evaluations reveal that even the strongest open model (GPT-OSS-120B) achieves below 60% accuracy in task completion and planning, with agents getting stuck in repetitive behaviors or failing to navigate basic obstacles. Since poor navigation confounds evaluation of social intelligence, SocialGrid offers an optional Planning Oracle to isolate social reasoning from planning deficits. While planning assistance improves task completion, social reasoning remains a bottleneck: agents fail to detect deception at near-random chance regardless of scale, relying on shallow heuristics rather than accumulating behavioral evidence. SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents. We also establish a competitive leaderboard using Elo ratings from adversarial league play.