LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
This work addresses a key obstacle for researchers and practitioners using joint embedding SSL methods by providing a more robust evaluation metric, though it is incremental as it builds on existing approaches.
The paper tackles the problem of evaluating learned representations in joint embedding SSL architectures without downstream tasks, introducing LiDAR as a metric that quantifies the rank of the LDA matrix to measure representation quality. The result shows that LiDAR significantly outperforms naive rank-based approaches in predicting optimal hyperparameters.
Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent challenge of evaluating learned representations without access to a downstream task, and an annotated dataset. Without efficient and reliable evaluation, it is difficult to iterate on architectural and training choices for JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the quality of representations within JE architectures. Our metric addresses several shortcomings of recent approaches based on feature covariance rank by discriminating between informative and uninformative features. In essence, LiDAR quantifies the rank of the Linear Discriminant Analysis (LDA) matrix associated with the surrogate SSL task -- a measure that intuitively captures the information content as it pertains to solving the SSL task. We empirically demonstrate that LiDAR significantly surpasses naive rank based approaches in its predictive power of optimal hyperparameters. Our proposed criterion presents a more robust and intuitive means of assessing the quality of representations within JE architectures, which we hope facilitates broader adoption of these powerful techniques in various domains.