A Random Matrix Theory of Masked Self-Supervised Regression
This provides theoretical insights into the mechanisms of masked SSL, a foundational training paradigm for transformers, which is incremental but offers precise analytical results.
The paper tackles the analysis of masked self-supervised learning (SSL) in high-dimensional settings, deriving explicit expressions for generalization error and showing that the joint predictor undergoes a phase transition, with masked SSL outperforming PCA in structured regimes.
In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor rather than a single vector-valued estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik--Ben Arous--Péché (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods