KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning
This work addresses the need for more flexible and stable self-supervised learning algorithms, though it appears incremental as it builds upon existing Joint-Embedding Predictive Architectures.
The authors tackled the problem of improving self-supervised learning by introducing KerJEPAs, a family of algorithms with kernel-based regularizers, which resulted in improved training stability and design flexibility compared to existing methods.
Recent breakthroughs in self-supervised Joint-Embedding Predictive Architectures (JEPAs) have established that regularizing Euclidean representations toward isotropic Gaussian priors yields provable gains in training stability and downstream generalization. We introduce a new, flexible family of KerJEPAs, self-supervised learning algorithms with kernel-based regularizers. One instance of this family corresponds to the recently-introduced LeJEPA Epps-Pulley regularizer which approximates a sliced maximum mean discrepancy (MMD) with a Gaussian prior and Gaussian kernel. By expanding the class of viable kernels and priors and computing the closed-form high-dimensional limit of sliced MMDs, we develop alternative KerJEPAs with a number of favorable properties including improved training stability and design flexibility.