Normalized Contrastive Learning for Text-Video Retrieval
This addresses a normalization issue in multimodal retrieval for researchers and practitioners, offering consistent gains but is incremental as it builds on existing contrastive learning frameworks.
The paper tackled the problem of incorrect normalization in cross-modal contrastive learning for text-video retrieval, which causes over- or under-representation of instances and hurts performance. The proposed Normalized Contrastive Learning (NCL) method achieved new state-of-the-art metrics on ActivityNet, MSVD, and MSR-VTT datasets.
Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval. Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new state-of-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering.