Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
For practitioners deploying multiple LLMs under cost and resource constraints, this work provides a practical routing solution that improves robustness and efficiency.
This paper proposes a batch-level, resource-aware routing framework for LLMs that jointly optimizes model assignment under cost and capacity constraints, including a robust variant for uncertainty. Experiments show robustness improves accuracy by 1-14%, batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields up to 3% additional gains.
We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further introduce a robust variant that accounts for uncertainty in predicted LLM performance, along with an offline instance allocation procedure that balances quality and throughput across multiple models. Experiments on two multi-task LLM benchmarks show that robustness improves accuracy by 1-14% over non-robust counterparts (depending on the performance estimator), batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields additional gains of up to 3% compared to a non-optimized allocation, all while strictly controlling cost and GPU resource constraints.