Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
This addresses a limitation in SSL for non-iconic data, offering incremental insights into optimizing view diversity for improved learning.
The paper tackles the problem of self-supervised learning (SSL) breaking down when instance consistency is not guaranteed, such as with non-iconic data, and finds that SSL can still learn meaningful representations, with moderate view diversity enhancing downstream performance on tasks like classification and dense prediction.
Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects or semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, we demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce effectiveness, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover's Distance (EMD) as an estimator to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate our findings across a range of settings, highlighting their robustness and applicability on diverse data sources.