Communicate-Predict-Act: Evaluating Social Intelligence of Agents
This work addresses the need for better evaluation of social intelligence in AI agents, which is crucial as they are deployed in real-world social settings, though it is incremental in refining existing assessment methods.
The authors tackled the problem of evaluating social intelligence in LLM agents by introducing a multiplayer social game arena and a COMPACT interaction protocol, finding that influence, transparency, and adaptability are more predictive of success than Theory of Mind or planning, with sociocognitive metrics achieving an AUC ROC of 0.82 in predicting game outcomes.
As large language model (LLM) agents become more prevalent in real world social settings, social intelligence will play an increasingly critical role. But social intelligence is still a poorly defined construct, for humans and artificial agents. We introduce a multiplayer arena of mixed cooperative and competitive social games to study LLM social intelligence. The controllability of LLM based agents enables systematic evaluation, which also supports broader inferences about social intelligence per se. We evaluated eight diverse LLMs (24B to 1T parameters) using a Communicate Predict Act (COMPACT) interaction protocol and fine grained probing of social dynamics. Elo style ratings reveal consistent performance differences across models, but this scalar measure provides only a partial characterization of social intelligence. To address this limitation, we analyze gameplay traces to extract sociocognitive metrics capturing action prediction, communicative influence, strategic reasoning, and tradeoffs under conflicting interests. These sociocognitive metrics exhibit strong intramodel consistency and they reliably predict pairwise agent advantage in game outcomes (AUC ROC = 0.82). Feature importance analysis indicates that surprisingly, influence, transparency, and adaptability are more predictive of success than Theory of Mind inference or deep planning. Together, our results advance a testable, multidimensional conception of social intelligence and provide empirical insights into the capacities that underpin it.