Leveraging Uncertainty Estimation for Efficient LLM Routing
This work addresses the need for balanced cost and quality in LLM deployment for edge-cloud systems, offering an incremental improvement over existing routing methods.
The paper tackles the problem of efficiently routing large language models (LLMs) in edge-cloud environments by proposing a confidence-driven router that uses uncertainty estimation, achieving superior response quality and cost efficiency in experiments on benchmarks like MT-Bench, GSM8K, and MMLU.
Deploying large language models (LLMs) in edge-cloud environments requires an efficient routing strategy to balance cost and response quality. Traditional approaches prioritize either human-preference data or accuracy metrics from benchmark datasets as routing criteria, but these methods suffer from rigidity and subjectivity. Moreover, existing routing frameworks primarily focus on accuracy and cost, neglecting response quality from a human preference perspective. In this work, we propose the Confidence-Driven LLM Router, a novel framework that leverages uncertainty estimation to optimize routing decisions. To comprehensively assess routing performance, we evaluate both system cost efficiency and response quality. In particular, we introduce the novel use of LLM-as-a-Judge to simulate human rating preferences, providing the first systematic assessment of response quality across different routing strategies. Extensive experiments on MT-Bench, GSM8K, and MMLU demonstrate that our approach outperforms state-of-the-art routing methods, achieving superior response quality while maintaining cost efficiency.