Fast Rates for Semi-Supervised Learning via Data-Augmentation Graph Regularization
Provides the first theoretical explanation for the labeled-sample efficiency of self-supervised learning, addressing a key gap for practitioners relying on few labels.
The paper proves that self-supervised learning with data augmentation achieves a fast transductive rate of O(1/n_L) in the number of labels, compared to the supervised rate of O(1/√n_L), by framing it as graph-Laplacian-regularized learning on an augmentation-induced similarity graph. The bound explicitly depends on augmentation quality via a data-augmentation alignment error.
Self-supervised learning matches supervised accuracy from a fraction of the labels, but the labeled-sample efficiency behind this has lacked a theoretical explanation. We provide one. Data augmentation induces a similarity graph on the unlabeled data, so downstream learning on that graph is graph-Laplacian-regularized learning. We prove a fast transductive rate, $O(1/n_L)$ in the number of labels, in place of the supervised $O(1/\sqrt{n_L})$, by carrying the leave-one-out stability apparatus of Johnson and Zhang (JMLR 2007) over to the augmentation graph, and without the unrealistic assumptions of limit-based analyses (exact kernel, generalizing features). The bound makes augmentation quality explicit: the expected error is at most $C/n_L + R_{\mathrm{DA}}(y)$, where the data-augmentation alignment error $R_{\mathrm{DA}}(y)$ is the graph-cut mass of augmentations that cross a label boundary, so good augmentations let few labels suffice. The analysis uses a streamlined loss that drops the projector, negative-sample, and orthogonality overhead of standard objectives yet still recovers the top-$K$ ideal features in the infinite-data limit, the augmentation-kernel eigenspace studied by Zhai et al. The result explains the observed accuracy-versus-label-count curve rather than only bounding a generalization gap.