Rethinking Deep Contrastive Learning with Embedding Memory
This work addresses the challenge of optimizing contrastive learning for real-world applications like visual retrieval, offering a simpler and more efficient approach, though it is incremental as it builds on existing memory-based methods.
The paper tackled the problem of designing effective pair-wise loss functions in deep metric learning by proposing a systematic methodology to study weighting strategies using embedding memory, and found that mining hard negatives while discarding easy ones is critical, leading to superior performance on large-scale visual retrieval benchmarks compared to recent mini-batch based methods.
Pair-wise loss functions have been extensively studied and shown to continuously improve the performance of deep metric learning (DML). However, they are primarily designed with intuition based on simple toy examples, and experimentally identifying the truly effective design is difficult in complicated, real-world cases. In this paper, we provide a new methodology for systematically studying weighting strategies of various pair-wise loss functions, and rethink pair weighting with an embedding memory. We delve into the weighting mechanisms by decomposing the pair-wise functions, and study positive and negative weights separately using direct weight assignment. This allows us to study various weighting functions deeply and systematically via weight curves, and identify a number of meaningful, comprehensive and insightful facts, which come up with our key observation on memory-based DML: it is critical to mine hard negatives and discard easy negatives which are less informative and redundant, but weighting on positive pairs is not helpful. This results in an efficient but surprisingly simple rule to design the weighting scheme, making it significantly different from existing mini-batch based methods which design various sophisticated loss functions to weight pairs carefully. Finally, we conduct extensive experiments on three large-scale visual retrieval benchmarks, and demonstrate the superiority of memory-based DML over recent mini-batch based approaches, by using a simple contrastive loss with momentum-updated memory.