HiddenBench: Assessing Collective Reasoning in Multi-Agent LLMs via Hidden Profile Tasks
This provides a reproducible benchmark for researchers to assess and improve collective reasoning in multi-agent LLMs, addressing a gap in systematic evaluation.
The authors tackled the problem of evaluating collective reasoning in multi-agent LLMs by introducing HiddenBench, a benchmark based on the Hidden Profile paradigm, and found that GPT-4.1 groups failed to integrate distributed knowledge, with some models like Gemini-2.5-Flash/Pro performing better but scale and reasoning not reliably indicating stronger collective reasoning.
Multi-agent systems built on large language models (LLMs) promise enhanced problem-solving through distributed information integration, but may also replicate collective reasoning failures observed in human groups. Yet the absence of a theory-grounded benchmark makes it difficult to systematically evaluate and improve such reasoning. We introduce HiddenBench, the first benchmark for evaluating collective reasoning in multi-agent LLMs. It builds on the Hidden Profile paradigm from social psychology, where individuals each hold asymmetric pieces of information and must communicate to reach the correct decision. To ground the benchmark, we formalize the paradigm with custom tasks and show that GPT-4.1 groups fail to integrate distributed knowledge, exhibiting human-like collective reasoning failures that persist even with varied prompting strategies. We then construct the full benchmark, spanning 65 tasks drawn from custom designs, prior human studies, and automatic generation. Evaluating 15 LLMs across four model families, HiddenBench exposes persistent limitations while also providing comparative insights: some models (e.g., Gemini-2.5-Flash/Pro) achieve higher performance, yet scale and reasoning are not reliable indicators of stronger collective reasoning. Our work delivers the first reproducible benchmark for collective reasoning in multi-agent LLMs, offering diagnostic insight and a foundation for future research on artificial collective intelligence.