Multi-agent decision making: A Blackwell's informativeness approach
Provides a principled framework for analyzing multi-LLM decision-making, offering a theoretically grounded method that improves performance over ad-hoc approaches.
The paper addresses the lack of formal guarantees in multi-agent decision-making with LLMs, showing that voting and debate are no more informative than pooled private information. They propose a method based on Blackwell's informativeness that outperforms existing methods on six QA benchmarks.
The rapid development of large language models (LLMs) has motivated research on decision-making in multi-agent systems, where multiple agents collaborate to achieve shared objectives. Existing aggregation approaches, such as voting and debate, are largely ad-hoc and lack formal guarantees regarding the informativeness of the resulting decisions. In this paper, we provide a principled approach to analyse decisions made in the multi-LLM setting using Blackwell's informativeness framework. Within the Blackwell information-structure abstraction, we show that voting and debate induce information structures that are no more informative than the pooled private information of all agents. This result identifies Bayesian pooled posterior maximisation as an information-theoretic upper-bound decision rule under the Blackwell ordering. Motivated by this theoretical analysis, we introduce a practical method for LLM-based question-answering (QA) tasks that estimates each agent's posterior and approximates the pooled posterior using a product-of-posteriors estimator. Extensive experiments on six QA benchmarks demonstrate that our approach outperforms state-of-the-art multi-LLM debate and voting methods.