Spectal Harmonics: Bridging Spectral Embedding and Matrix Completion in Self-Supervised Learning
It offers a theoretical foundation for self-supervised learning, which is incremental in building a coherent understanding of existing methods.
The paper tackles the problem of understanding self-supervised learning methods by connecting them to a low-rank matrix completion problem, providing theoretical analysis on convergence and downstream performance.
Self-supervised methods received tremendous attention thanks to their seemingly heuristic approach to learning representations that respect the semantics of the data without any apparent supervision in the form of labels. A growing body of literature is already being published in an attempt to build a coherent and theoretically grounded understanding of the workings of a zoo of losses used in modern self-supervised representation learning methods. In this paper, we attempt to provide an understanding from the perspective of a Laplace operator and connect the inductive bias stemming from the augmentation process to a low-rank matrix completion problem. To this end, we leverage the results from low-rank matrix completion to provide theoretical analysis on the convergence of modern SSL methods and a key property that affects their downstream performance.