Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning
This provides theoretical insights into SSL effectiveness, which is incremental for researchers in machine learning.
The paper tackles the problem of understanding why reconstructive self-supervised learning (SSL) works by analyzing its data-generating mechanism, showing that perfect linear approximation requires a full-rank component preserving labels and a redundant component approximated via low-rank factorization, with theoretical risk analysis and experimental validation.
We study the data-generating mechanism for reconstructive SSL to shed light on its effectiveness. With an infinite amount of labeled samples, we provide a sufficient and necessary condition for perfect linear approximation. The condition reveals a full-rank component that preserves the label classes of Y, along with a redundant component. Motivated by the condition, we propose to approximate the redundant component by a low-rank factorization and measure the approximation quality by introducing a new quantity $ε_s$, parameterized by the rank of factorization s. We incorporate $ε_s$ into the excess risk analysis under both linear regression and ridge regression settings, where the latter regularization approach is to handle scenarios when the dimension of the learned features is much larger than the number of labeled samples n for downstream tasks. We design three stylized experiments to compare SSL with supervised learning under different settings to support our theoretical findings.