Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
This work addresses generalization challenges in SSL for computer vision, offering incremental improvements through a novel embedding technique.
The paper tackles the problem of improving generalization in self-supervised learning by introducing Simplicial Embeddings (SEM), which project representations into simplices to impose group sparsity, and shows that this leads to better generalization on datasets like CIFAR-100 and ImageNet, with emergent semantic coherence in features.
Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a softmax operation. This procedure conditions the representation onto a constrained space during pretraining and imparts an inductive bias for group sparsity. For downstream classification, we formally prove that the SEM representation leads to better generalization than an unnormalized representation. Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and ImageNet. Finally, when used in a downstream classification task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.