Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning
This addresses cost efficiency for scalable multi-agent LLM systems, though it is incremental as it builds on existing multi-agent and RL concepts.
The paper tackles the problem of high and uncontrolled inference costs in multi-agent LLM systems by introducing a centralized framework with a controller LLM that selectively coordinates expert models using reinforcement learning to optimize performance-cost trade-offs. Experiments on four benchmarks show the system surpasses the best expert LLM under high budgets and maintains strong performance in low-budget modes.
Large language models (LLMs) exhibit complementary strengths across domains and come with varying inference costs, motivating the design of multi-agent LLM systems where specialized models collaborate efficiently. Existing approaches predominantly rely on decentralized frameworks, which invoke multiple LLMs for every input and thus lead to substantial and uncontrolled inference costs. In this work, we introduce a centralized multi-LLM framework, where a controller LLM selectively coordinates a pool of expert models in a cost-efficient and cost-controllable manner. We formulate this coordination problem as reinforcement learning with dual objectives: maximizing task performance while minimizing the overall inference cost. In addition, we expect the multi-agent system to have adapted behavior with different budget conditions during inference. To this end, we propose CoRL, a reinforcement learning framework that optimizes the performance cost trade-off in a controllable multi-budget setting. Experiments on four diverse benchmarks demonstrate that CoRL enables a single system to surpass the best expert LLM under high-budget settings, while maintaining strong performance in more economical low-budget modes, highlighting the effectiveness of centralized coordination for scalable and cost-efficient multi-agent LLM systems.