Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus
For practitioners building multi-agent LLM systems, this work reveals a critical failure mode and offers a lightweight fix, though the gains are modest and task-specific.
The paper identifies representational collapse in multi-agent LLM committees, where agents produce nearly identical reasoning despite different role prompts. The proposed DALC protocol achieves 87% accuracy on GSM8K, outperforming self-consistency (84%) with 26% lower token cost.
Multi-agent LLM committees replicate the same model under different role prompts and aggregate outputs by majority vote, implicitly assuming that agents contribute complementary evidence. We embed each agent's chain-of-thought rationale and measure pairwise similarity: across 100 GSM8K questions with three Qwen2.5-14B agents, mean cosine similarity is 0.888 and effective rank is 2.17 out of 3.0, a failure mode we term representational collapse. DALC, a training-free consensus protocol that computes diversity weights from embedding geometry, reaches 87% on GSM8K versus 84% for self-consistency at 26% lower token cost. Ablation experiments reveal 1-3 point per-protocol run-to-run variance, confirm that hint sharing contributes more than diversity weighting alone, and show that encoder choice strongly modulates collapse severity (cosine 0.908 with mxbai versus 0.888 with nomic) and downstream accuracy. The more robust finding is that collapse is measurable, worsens on harder tasks, and that the choice of embedding proxy is a first-order design decision for any latent communication protocol.