SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context
This addresses the problem of balancing performance and cost in KG-RAG systems for users deploying large language models, offering a plug-and-play solution that is incremental as it builds on existing routing concepts but adapts them specifically to RAG.
The paper tackles the high inference costs in knowledge graph retrieval-augmented generation (KG-RAG) by proposing SkewRoute, a training-free LLM routing method that uses score skewness from retrieval to direct queries to appropriate LLMs, achieving over 3x higher routing effectiveness and reducing runtime to less than 0.001x compared to existing methods.
Large language models excel at many tasks but often incur high inference costs during deployment. To mitigate hallucination, many systems use a knowledge graph to enhance retrieval-augmented generation (KG-RAG). However, the large amount of retrieved knowledge contexts increase these inference costs further. A promising solution to balance performance and cost is LLM routing, which directs simple queries to smaller LLMs and complex ones to larger LLMs. However, no dedicated routing methods currently exist for RAG, and existing training-based routers face challenges scaling to this domain due to the need for extensive training data. We observe that the score distributions produced by the retrieval scorer strongly correlate with query difficulty. Based on this, we propose an extremely simple yet effective routing framework, the first specifically designed for KG-RAG that efficiently balances performance and cost in a plug-and-play manner. It delivers over 3x higher routing effectiveness while reducing runtime to less than 0.001x compared to existing methods. Our code is available at https://github.com/hrwang00/SkewRoute.