RouterRetriever: Routing over a Mixture of Expert Embedding Models
This addresses the issue of retrieval accuracy in specialized domains for users needing efficient, adaptable systems, representing a novel application of routing mechanisms in retrieval rather than an incremental improvement.
The paper tackles the problem of information retrieval underperformance in domain-specific tasks by introducing RouterRetriever, a model that routes queries to domain-specific expert retrievers, resulting in a 2.1 absolute nDCG@10 improvement over general models and 3.2 over multi-task models on the BEIR benchmark.
Information retrieval methods often rely on a single embedding model trained on large, general-domain datasets like MSMARCO. While this approach can produce a retriever with reasonable overall performance, they often underperform models trained on domain-specific data when testing on their respective domains. Prior work in information retrieval has tackled this through multi-task training, but the idea of routing over a mixture of domain-specific expert retrievers remains unexplored despite the popularity of such ideas in language model generation research. In this work, we introduce RouterRetriever, a retrieval model that leverages a mixture of domain-specific experts by using a routing mechanism to select the most appropriate expert for each query. RouterRetriever is lightweight and allows easy addition or removal of experts without additional training. Evaluation on the BEIR benchmark demonstrates that RouterRetriever outperforms both models trained on MSMARCO (+2.1 absolute nDCG@10) and multi-task models (+3.2). This is achieved by employing our routing mechanism, which surpasses other routing techniques (+1.8 on average) commonly used in language modeling. Furthermore, the benefit generalizes well to other datasets, even in the absence of a specific expert on the dataset. RouterRetriever is the first work to demonstrate the advantages of routing over a mixture of domain-specific expert embedding models as an alternative to a single, general-purpose embedding model, especially when retrieving from diverse, specialized domains.