CascadeDebate: Multi-Agent Deliberation for Cost-Aware LLM Cascades
For practitioners deploying LLM cascades, CascadeDebate offers a method to dynamically balance accuracy and cost by resolving uncertainty with lightweight agent ensembles before escalating to expensive models or human experts.
CascadeDebate inserts multi-agent deliberation at escalation boundaries of LLM cascades, enabling cost-aware resolution of ambiguous queries without premature escalation. It achieves up to 26.75% improvement over strong baselines across five benchmarks.
Cascaded LLM systems coordinate models of varying sizes with human experts to balance accuracy, cost, and abstention under uncertainty. However, single-model tiers at each stage often struggle with ambiguous queries, triggering premature escalations to costlier models or experts due to under-confidence and inefficient compute scaling. CascadeDebate addresses this gap by inserting multi-agent deliberation directly at each tier's escalation boundary. Confidence-based routers activate lightweight agent ensembles only for uncertain cases, enabling consensus-driven resolution of ambiguities internally without invoking higher-cost upgrades. Our unified architecture alternates single-model inference with selective multi-agent deliberation across model scales, culminating in human experts as the final fallback. This design scales test-time compute dynamically according to query difficulty. Across five benchmarks spanning science, medicine, and general knowledge, CascadeDebate outperforms strong single-model cascades and standalone multi-agent systems by up to 26.75 percent. An online threshold optimizer proves essential, boosting accuracy by 20.98 to 52.33 percent relative improvement over fixed policies and enabling elastic adaptation to real-world distributions.