Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation
This addresses the efficiency problem for large-scale recommender systems in industrial settings, offering a systematic approach to candidate generation, though it is incremental in building on existing two-stage schemes.
The paper tackles the computational inefficiency of generating Top-N recommendations from large corpora by proposing CIGAR, a candidate generation and re-ranking framework that uses binary embeddings for efficient retrieval and real-valued models for ranking, resulting in significant accuracy improvements and orders of magnitude reduction in query time on large-scale datasets with millions of users/items.
Generating the Top-N recommendations from a large corpus is computationally expensive to perform at scale. Candidate generation and re-ranking based approaches are often adopted in industrial settings to alleviate efficiency problems. However it remains to be fully studied how well such schemes approximate complete rankings (or how many candidates are required to achieve a good approximation), or to develop systematic approaches to generate high-quality candidates efficiently. In this paper, we seek to investigate these questions via proposing a candidate generation and re-ranking based framework (CIGAR), which first learns a preference-preserving binary embedding for building a hash table to retrieve candidates, and then learns to re-rank the candidates using real-valued ranking models with a candidate-oriented objective. We perform a comprehensive study on several large-scale real-world datasets consisting of millions of users/items and hundreds of millions of interactions. Our results show that CIGAR significantly boosts the Top-N accuracy against state-of-the-art recommendation models, while reducing the query time by orders of magnitude. We hope that this work could draw more attention to the candidate generation problem in recommender systems.