DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles
This addresses uncertainty quantification for LLM agent ensembles, offering a more reliable method for complex reasoning tasks, though it is incremental as it builds on existing multi-agent systems.
The paper tackled the problem of quantifying uncertainty in multi-agent LLM systems by introducing DiscoUQ, a framework that analyzes structured disagreement among agents, resulting in improved calibration and performance, with DiscoUQ-LLM achieving an average AUROC of 0.802 and ECE of 0.036, outperforming baselines.
Multi-agent LLM systems, where multiple prompted instances of a language model independently answer questions, are increasingly used for complex reasoning tasks. However, existing methods for quantifying the uncertainty of their collective outputs rely on shallow voting statistics that discard the rich semantic information in agents' reasoning. We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement -- both linguistic properties (evidence overlap, argument strength, divergence depth) and embedding geometry (cluster distances, dispersion, cohesion) -- to produce well-calibrated confidence estimates. We propose three methods of increasing complexity: DiscoUQ-LLM (logistic regression on LLM-extracted structure features), DiscoUQ-Embed (logistic regression on embedding geometry), and DiscoUQ-Learn (a neural network combining all features). Evaluated on four diverse benchmarks (StrategyQA, MMLU, TruthfulQA, ARC-Challenge) with a 5-agent system using Qwen3.5-27B, DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline (LLM Aggregator, 0.791) while being substantially better calibrated (ECE 0.036 vs. 0.098). The learned features generalize across benchmarks with near-zero performance degradation and provide the largest improvements where they are most needed: in the ambiguous "weak disagreement" tier where simple vote counting fails.