ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outliers
This addresses a domain-specific issue for AI platforms using LLM routers, offering an incremental improvement in handling outliers.
The paper tackles the problem of nonparametric LLM query routers struggling with outlier queries due to limited training diversity, proposing ProxRouter to improve robustness. Experiments show it enhances outlier routing while maintaining inlier performance with minimal overhead.
Large language model (LLM) query routers are critical to modern AI platforms as they seek to improve efficiency by assigning inference queries to accurate, yet low-cost models. Parametric routers typically use trained neural networks for LLM selection but suffer from retraining and maintenance overheads. Nonparametric routers are training-free, instead estimating LLM accuracy and cost via similarity between encodings of the input query and training set queries. However, like their parametric counterparts, nonparametric routers struggle to generalize to outlier queries, an issue exacerbated by limited diversity in training sets which are costly to expand and difficult to keep current with ever-evolving use cases. We propose ProxRouter, which applies an exponentially tilted aggregation mechanism to balance bias and variance in nonparametric routers, improving their robustness to outliers. Experiments show ProxRouter enhances outlier routing while preserving inlier performance with minimal overhead.