Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections
This work addresses the need for benchmarks to assess social intelligence in AI agents, particularly for collaborative tasks, though it is incremental as it builds on existing game-based evaluation methods.
The authors introduced the improvisational wordplay game Connections as a benchmark to test social intelligence in AI agents, requiring skills like knowledge retrieval and gauging other agents' understanding, and demonstrated its effectiveness in evaluating collaborative reasoning beyond individual memory and deduction.
We formally introduce a improvisational wordplay game called Connections to explore reasoning capabilities of AI agents. Playing Connections combines skills in knowledge retrieval, summarization and awareness of cognitive states of other agents. We show how the game serves as a good benchmark for social intelligence abilities of language model based agents that go beyond the agents' own memory and deductive reasoning and also involve gauging the understanding capabilities of other agents. Finally, we show how through communication with other agents in a constrained environment, AI agents must demonstrate social awareness and intelligence in games involving collaboration.