Kernel VICReg for Self-Supervised Learning in Reproducing Kernel Hilbert Space
This work addresses the problem of capturing nonlinear dependencies in self-supervised learning for representation learning, offering a novel approach that bridges kernel methods with modern SSL, though it appears incremental as it builds on the existing VICReg framework.
The paper tackled the limitation of self-supervised learning methods operating in Euclidean space by proposing Kernel VICReg, which lifts the VICReg objective into a Reproducing Kernel Hilbert Space, resulting in consistent performance gains across datasets like MNIST, CIFAR-10, and ImageNet100, with strong improvements on data with nonlinear structures.
Self-supervised learning (SSL) has emerged as a powerful paradigm for representation learning by optimizing geometric objectives--such as invariance to augmentations, variance preservation, and feature decorrelation--without requiring labels. However, most existing methods operate in Euclidean space, limiting their ability to capture nonlinear dependencies and geometric structures. In this work, we propose Kernel VICReg, a novel self-supervised learning framework that lifts the VICReg objective into a Reproducing Kernel Hilbert Space (RKHS). By kernelizing each term of the loss-variance, invariance, and covariance--we obtain a general formulation that operates on double-centered kernel matrices and Hilbert-Schmidt norms, enabling nonlinear feature learning without explicit mappings. We demonstrate that Kernel VICReg not only avoids representational collapse but also improves performance on tasks with complex or small-scale data. Empirical evaluations across MNIST, CIFAR-10, STL-10, TinyImageNet, and ImageNet100 show consistent gains over Euclidean VICReg, with particularly strong improvements on datasets where nonlinear structures are prominent. UMAP visualizations further confirm that kernel-based embeddings exhibit better isometry and class separation. Our results suggest that kernelizing SSL objectives is a promising direction for bridging classical kernel methods with modern representation learning.