Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization
This work addresses a foundational problem for researchers in machine learning and AI by bridging the gap between empirical success and theoretical understanding in representation learning, though it appears incremental as an overview with contributions.
The paper tackles the challenge of analyzing deep learning models for unsupervised representation learning, which use new principles like self-supervision that are not easily explained by classical theories, and it provides an overview of recent theoretical advances combining statistics and optimization to characterize these representations.
Representation learning from unlabeled data has been extensively studied in statistics, data science and signal processing with a rich literature on techniques for dimension reduction, compression, multi-dimensional scaling among others. However, current deep learning models use new principles for unsupervised representation learning that cannot be easily analyzed using classical theories. For example, visual foundation models have found tremendous success using self-supervision or denoising/masked autoencoders, which effectively learn representations from massive amounts of unlabeled data. However, it remains difficult to characterize the representations learned by these models and to explain why they perform well for diverse prediction tasks or show emergent behavior. To answer these questions, one needs to combine mathematical tools from statistics and optimization. This paper provides an overview of recent theoretical advances in representation learning from unlabeled data and mentions our contributions in this direction.