Relevance-Based Embeddings: Lightweight Candidate Retrieval via Heavy-Ranker Calls
This work addresses the efficiency-accuracy trade-off in candidate retrieval for large-scale search systems, offering a theoretically grounded alternative to traditional embedding methods.
The paper proposes a method to improve candidate retrieval by using scores from an expensive ranking model to create relevance-based embeddings for queries and items. The approach is theoretically shown to approximate any complex similarity model and achieves strong empirical results on academic and production datasets.
In many machine learning applications, the most relevant items for a query should be efficiently retrieved. The relevance function is usually an expensive similarity model, making the exhaustive search infeasible. A typical solution is to train another model that separately embeds queries and items to a vector space, where similarity is defined via the dot product or cosine similarity. This allows one to search the relevant items through fast approximate nearest neighbor search at the cost of some reduction in quality. To compensate for this reduction, the found items (candidates) are re-ranked by the expensive ranking model. In this paper, we investigate an alternative approach to candidate selection that utilizes the scores of the expensive model to improve the representations of queries and items. The idea is to describe each query (item) by its relevance to a set of support items (queries) and use these new representations to obtain query (item) embeddings. We theoretically prove that such embeddings are powerful enough to approximate any complex similarity model (under mild conditions). We also investigate the choice of support items, which is a crucial ingredient of the proposed approach. The experiments on diverse academic and production datasets illustrate the power of our method.