Homomorphic Self-Supervised Learning
This work provides a new theoretical perspective for researchers in self-supervised learning, but it appears incremental as it builds on existing equivariance concepts without demonstrating broad practical gains.
The paper tackles the problem of unifying self-supervised learning algorithms by introducing a framework based on equivariant representations, showing theoretically that it can subsume input augmentations and validating this experimentally for simple cases.
In this work, we observe that many existing self-supervised learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically, we introduce a general framework we call Homomorphic Self-Supervised Learning, and theoretically show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate how the framework fails when representational structure is removed, and further empirically explore how the parameters of this framework relate to those of traditional augmentation-based self-supervised learning. We conclude with a discussion of the potential benefits afforded by this new perspective on self-supervised learning.