What Should Not Be Contrastive in Contrastive Learning
This addresses the limitation of existing contrastive methods for computer vision by improving adaptability to downstream tasks without prior invariance knowledge, though it is incremental as it builds on current frameworks.
The paper tackles the problem of self-supervised contrastive learning assuming fixed invariances that may harm downstream task performance, and introduces a framework that learns separate embedding spaces for varying and invariant factors, achieving best performance across diverse classification tasks and data corruptions.
Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations. However, these methods implicitly assume a particular set of representational invariances (e.g., invariance to color), and can perform poorly when a downstream task violates this assumption (e.g., distinguishing red vs. yellow cars). We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but one augmentation. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks. We further find that the concatenation of the invariant and varying spaces performs best across all tasks we investigate, including coarse-grained, fine-grained, and few-shot downstream classification tasks, and various data corruptions.