EnSiam: Self-Supervised Learning With Ensemble Representations
This addresses a stability issue in self-supervised learning for researchers and practitioners, but it is incremental as it builds on existing contrastive learning frameworks.
The paper tackles the sensitivity of SimSiam to training configurations in self-supervised learning by proposing EnSiam, which uses ensemble representations to provide stable pseudo labels, resulting in improved performance that outperforms previous state-of-the-art methods on datasets like ImageNet.
Recently, contrastive self-supervised learning, where the proximity of representations is determined based on the identities of samples, has made remarkable progress in unsupervised representation learning. SimSiam is a well-known example in this area, known for its simplicity yet powerful performance. However, it is known to be sensitive to changes in training configurations, such as hyperparameters and augmentation settings, due to its structural characteristics. To address this issue, we focus on the similarity between contrastive learning and the teacher-student framework in knowledge distillation. Inspired by the ensemble-based knowledge distillation approach, the proposed method, EnSiam, aims to improve the contrastive learning procedure using ensemble representations. This can provide stable pseudo labels, providing better performance. Experiments demonstrate that EnSiam outperforms previous state-of-the-art methods in most cases, including the experiments on ImageNet, which shows that EnSiam is capable of learning high-quality representations.