CIPER: Combining Invariant and Equivariant Representations Using Contrastive and Predictive Learning
This work addresses a problem in computer vision for researchers and practitioners by improving self-supervised learning, though it appears incremental as it builds on existing contrastive and predictive methods.
The paper tackles the limitation of self-supervised representation learning methods that learn only invariant or equivariant features, which may not match downstream task requirements, by introducing CIPER, which combines both types of features using contrastive and predictive learning. The results show that CIPER outperforms a baseline contrastive method on various tasks and encourages hierarchically structured representations.
Self-supervised representation learning (SSRL) methods have shown great success in computer vision. In recent studies, augmentation-based contrastive learning methods have been proposed for learning representations that are invariant or equivariant to pre-defined data augmentation operations. However, invariant or equivariant features favor only specific downstream tasks depending on the augmentations chosen. They may result in poor performance when the learned representation does not match task requirements. Here, we consider an active observer that can manipulate views of an object and has knowledge of the action(s) that generated each view. We introduce Contrastive Invariant and Predictive Equivariant Representation learning (CIPER). CIPER comprises both invariant and equivariant learning objectives using one shared encoder and two different output heads on top of the encoder. One output head is a projection head with a state-of-the-art contrastive objective to encourage invariance to augmentations. The other is a prediction head estimating the augmentation parameters, capturing equivariant features. Both heads are discarded after training and only the encoder is used for downstream tasks. We evaluate our method on static image tasks and time-augmented image datasets. Our results show that CIPER outperforms a baseline contrastive method on various tasks. Interestingly, CIPER encourages the formation of hierarchically structured representations where different views of an object become systematically organized in the latent representation space.