Self-Supervised Learning Using Nonlinear Dependence
This addresses the limitation of current SSL methods for handling complex data like high-dimensional visual data, though it appears incremental as it extends existing paradigms.
The paper tackles the problem that existing self-supervised learning methods neglect nonlinear dependencies in complex data by introducing CDSSL, a framework that integrates both linear correlations and nonlinear dependencies using HSIC, resulting in improved representation quality on diverse benchmarks.
Self-supervised learning has gained significant attention in contemporary applications, particularly due to the scarcity of labeled data. While existing SSL methodologies primarily address feature variance and linear correlations, they often neglect the intricate relations between samples and the nonlinear dependencies inherent in complex data--especially prevalent in high-dimensional visual data. In this paper, we introduce Correlation-Dependence Self-Supervised Learning (CDSSL), a novel framework that unifies and extends existing SSL paradigms by integrating both linear correlations and nonlinear dependencies, encapsulating sample-wise and feature-wise interactions. Our approach incorporates the Hilbert-Schmidt Independence Criterion (HSIC) to robustly capture nonlinear dependencies within a Reproducing Kernel Hilbert Space, enriching representation learning. Experimental evaluations on diverse benchmarks demonstrate the efficacy of CDSSL in improving representation quality.