From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning
For researchers in self-supervised learning, this paper provides a conceptual framework and taxonomy that highlights predictive representation learning as a promising direction, though the experimental results are incremental and limited to three methods.
This paper introduces Predictive Representation Learning (PRL) as a new category in self-supervised learning, proposing a taxonomy that includes alignment, reconstruction, and PRL. Comparative experiments show MAE achieves perfect similarity (1.00) but low robustness (0.55), while BYOL and I-JEPA achieve accuracies of 0.98 and 0.95 with robustness scores of 0.75 and 0.78, respectively.
Self-supervised learning has emerged as a major technique for the task of learning from unlabeled data, where the current methods mostly revolve around alignment of representations and input recon struction. Although such approaches have demonstrated excellent performance in practice, their scope remains mostly confined to learning from observed data and does not provide much help in terms of a learning structure that is predictive of the data distribution. In this paper, we study some of the recent developments in the realm of self-supervised learning. We define a new category called Predictive Representation Learning (PRL), which revolves around the latent prediction of unobserved components of data based on the observation. We propose a common taxonomy that classifies PRL along with alignment and reconstruction-based learning approaches. Furthermore, we argue that Joint-Embedding Predictive Architecture(JEPA) can be considered as an exemplary member of this new paradigm. We further discuss theoretical perspectives and open challenges, highlighting predictive representation learning as a promising direction for future self-supervised learning research. In this study, we implemented Bootstrap Your Own Latent (BYOL), Masked Autoencoders (MAE), and Image-JEPA (I-JEPA) for comparative analysis. The results indicate that MAE achieves perfect similarity of 1.00, but exhibits relatively weak robustness of 0.55. In contrast, BYOL and I-JEPA attain accuracies of 0.98 and 0.95, with robustness scores of 0.75 and 0.78, respectively.