Self-Supervised Representation Learning via Hyperspherical Density Shaping
For researchers in self-supervised learning, HyDeS offers a theoretically grounded alternative to empirical heuristics, though its performance is mixed and incremental.
HyDeS proposes a theoretically grounded self-supervised learning method using hyperspherical density shaping, achieving strong performance on segmentation tasks like VOC PASCAL but lagging in fine-grained classification.
Modern self-supervised representation learning methods often relies on empirical heuristics that are not theoretically grounded. In this study we propose HyDeS, a theoretically grounded method based on multi-view mutual information maximization within an hyperspherical space using Shannon differential entropy with a non-parametric von Mises-Fisher density estimator. We show that HyDeS bias the trained model towards focusing on foreground features of the images and perform well on segmentation tasks such as VOC PASCAL, while it lags in fine-grained classification. We provide a detailed analysis of the induced latent space geometry and learning dynamics, that can be used for designing other theoretically grounded self-supervised learning methods.