Invariance-adapted decomposition and Lasso-type contrastive learning
This work provides a theoretical framework for analyzing contrastive learning mechanisms, which is incremental but offers insights for researchers in representation learning.
The paper tackles the problem of understanding the data structures captured by contrastive learning by introducing an invariance-adapted latent space decomposition that generalizes prior work, and experimentally shows that contrastive learning with a lasso-type metric can find this space, suggesting new potential for the method.
Recent years have witnessed the effectiveness of contrastive learning in obtaining the representation of dataset that is useful in interpretation and downstream tasks. However, the mechanism that describes this effectiveness have not been thoroughly analyzed, and many studies have been conducted to investigate the data structures captured by contrastive learning. In particular, the recent study of \citet{content_isolate} has shown that contrastive learning is capable of decomposing the data space into the space that is invariant to all augmentations and its complement. In this paper, we introduce the notion of invariance-adapted latent space that decomposes the data space into the intersections of the invariant spaces of each augmentation and their complements. This decomposition generalizes the one introduced in \citet{content_isolate}, and describes a structure that is analogous to the frequencies in the harmonic analysis of a group. We experimentally show that contrastive learning with lasso-type metric can be used to find an invariance-adapted latent space, thereby suggesting a new potential for the contrastive learning. We also investigate when such a latent space can be identified up to mixings within each component.