Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework
This work addresses the need for more effective and robust self-supervised representation learning methods in machine learning, though it appears incremental as it builds on existing contrastive learning and NCA concepts.
The authors tackled the problem of improving contrastive learning by connecting it to neighborhood component analysis, proposing a new framework that yields integrated contrastive losses, resulting in up to 6% improvement in standard accuracy and 17% in robust accuracy on downstream tasks.
As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samples for representation learning, which relates to exploiting neighborhood information in a feature space. By investigating the connection between contrastive learning and neighborhood component analysis (NCA), we provide a novel stochastic nearest neighbor viewpoint of contrastive learning and subsequently propose a series of contrastive losses that outperform the existing ones. Under our proposed framework, we show a new methodology to design integrated contrastive losses that could simultaneously achieve good accuracy and robustness on downstream tasks. With the integrated framework, we achieve up to 6\% improvement on the standard accuracy and 17\% improvement on the robust accuracy.