Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs
For multi-agent LLM systems, this work addresses the bottleneck of one-shot routing by enabling iterative refinement, significantly improving performance with reduced reliance on the strongest agent.
Existing multi-agent LLM controllers use one-shot routing without iterative refinement. The authors propose a critique-and-routing controller that formulates coordination as an MDP, optimizing via policy gradients, and achieves SOTA on seven reasoning benchmarks while using the strongest agent for fewer than 25% of calls.
Multi-agent large language model (LLM) systems often rely on a controller to coordinate a pool of heterogeneous models, yet existing controllers are typically limited to one-shot routing: they select a model once and return its output directly. Such routing-only designs provide no mechanism to critique intermediate drafts or support iterative refinement. To address this limitation, we propose a critique-and-routing controller that casts multi-agent coordination as a sequential decision problem. At each turn, the controller evaluates the current draft, decides whether to stop or continue, and, if needed, selects the next agent for further refinement. We formulate this process as a finite-horizon Markov Decision Process (MDP) with explicit agent-utilization constraints, design a composite reward for controller decisions across turns, and optimize the controller via policy gradients under a Lagrangian-relaxed objective. Extensive experiments across multiple heterogeneous multi-agent systems and seven reasoning benchmarks show that our method consistently outperforms state-of-the-art baselines and substantially narrows the gap to the strongest agent, while using it for fewer than 25% of total calls.