A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learning
This provides a theoretical foundation for principled augmentation design in self-supervised learning, addressing a key bottleneck for researchers and practitioners in machine learning.
The paper tackles the problem of designing optimal data augmentations for self-supervised learning by using kernel theory to derive analytical expressions for augmentations that achieve desired target representations, showing that augmentations need not be similar to the data or diverse and that architecture significantly impacts optimal choices.
Data augmentations play an important role in the recent success of self-supervised learning (SSL). While augmentations are commonly understood to encode invariances between different views into the learned representations, this interpretation overlooks the impact of the pretraining architecture and suggests that SSL would require diverse augmentations which resemble the data to work well. However, these assumptions do not align with empirical evidence, encouraging further theoretical understanding to guide the principled design of augmentations in new domains. To this end, we use kernel theory to derive analytical expressions for data augmentations that achieve desired target representations after pretraining. We consider non-contrastive and contrastive losses, namely VICReg, Barlow Twins and the Spectral Contrastive Loss, and provide an algorithm to construct such augmentations. Our analysis shows that augmentations need not be similar to the data to learn useful representations, nor be diverse, and that the architecture has a significant impact on the optimal augmentations.