RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers
This addresses the problem of choosing the right LLM router for developers and researchers, but it is incremental as it provides a benchmarking tool rather than a new routing method.
The paper tackles the challenge of selecting optimal LLM routers by introducing RouterArena, an open platform for comprehensive comparison, resulting in an initial leaderboard with detailed metrics.
Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances. However, the rapid emergence of various routers makes choosing the right one increasingly challenging. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for leaderboard updates. Leveraging our framework, we have produced the initial leaderboard with detailed metrics comparison as shown in Figure 1. Our framework for evaluating new routers is on https://github.com/RouteWorks/RouterArena