LLM Routing with Dueling Feedback
This addresses efficient LLM routing for users and developers by reducing costs and improving performance, though it appears incremental as it builds on existing bandit and embedding techniques.
The paper tackles the problem of selecting the best large language model (LLM) for each query by balancing user satisfaction, model expertise, and inference cost, formulating it as contextual dueling bandits to learn from pairwise feedback. Their method achieves lower cumulative regret and faster convergence with better robustness and performance-cost balance than strong baselines on RouterBench and MixInstruct datasets.
We study LLM routing, the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost. We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores, thereby yielding label-efficient and dynamic adaptation. Building on this formulation, we introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting. These embeddings enable the practical instantiation of Feel-Good Thompson Sampling for Contextual Dueling Bandits (FGTS.CDB), a theoretically grounded posterior-sampling algorithm. We propose four variants of the categorical weighting that explicitly integrate model quality and cost, and we empirically evaluate the proposed methods on the RouterBench and MixInstruct datasets. Across both benchmarks, our methods achieve lower cumulative regret and faster convergence, with better robustness and performance-cost balance than strong baselines built with a general-purpose OpenAI embedding model.