Cross-Batch Negative Sampling for Training Two-Tower Recommenders
This addresses a computational bottleneck for large-scale recommender systems, offering an incremental improvement over existing in-batch negative sampling methods.
The paper tackles the inefficiency of training two-tower recommender models with large batch sizes by proposing Cross-Batch Negative Sampling (CBNS), which uses encoded item embeddings from recent mini-batches to improve training, resulting in demonstrated effectiveness and efficiency in empirical evaluations.
The two-tower architecture has been widely applied for learning item and user representations, which is important for large-scale recommender systems. Many two-tower models are trained using various in-batch negative sampling strategies, where the effects of such strategies inherently rely on the size of mini-batches. However, training two-tower models with a large batch size is inefficient, as it demands a large volume of memory for item and user contents and consumes a lot of time for feature encoding. Interestingly, we find that neural encoders can output relatively stable features for the same input after warming up in the training process. Based on such facts, we propose a simple yet effective sampling strategy called Cross-Batch Negative Sampling (CBNS), which takes advantage of the encoded item embeddings from recent mini-batches to boost the model training. Both theoretical analysis and empirical evaluations demonstrate the effectiveness and the efficiency of CBNS.