Self-Supervised Learning for Group Equivariant Neural Networks
This work addresses a specific challenge in self-supervised learning for group equivariant neural networks, which are important for applications requiring symmetry handling, but it is incremental as it builds on existing equivariant network frameworks.
The paper tackles the problem of constructing pretext tasks for self-supervised learning in group equivariant neural networks, proposing equivariant pretext labels and invariant contrastive loss to ensure training consistency with equivariance, with experiments showing that these tasks are effectively exploited by the networks on standard image recognition benchmarks.
This paper proposes a method to construct pretext tasks for self-supervised learning on group equivariant neural networks. Group equivariant neural networks are the models whose structure is restricted to commute with the transformations on the input. Therefore, it is important to construct pretext tasks for self-supervised learning that do not contradict this equivariance. To ensure that training is consistent with the equivariance, we propose two concepts for self-supervised tasks: equivariant pretext labels and invariant contrastive loss. Equivariant pretext labels use a set of labels on which we can define the transformations that correspond to the input change. Invariant contrastive loss uses a modified contrastive loss that absorbs the effect of transformations on each input. Experiments on standard image recognition benchmarks demonstrate that the equivariant neural networks exploit the proposed equivariant self-supervised tasks.