Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guarantees
This addresses a key challenge in representation learning for machine learning practitioners by improving transferability and few-shot learning, though it appears incremental as it builds on existing self-supervised approaches.
The paper tackles the problem of biased sample risk in self-supervised learning methods that align covariance matrices, introducing Adv-SSL to achieve unbiased transfer learning without extra cost, and it outperforms existing methods on multiple benchmark datasets with theoretical guarantees.
Learning transferable data representations from abundant unlabeled data remains a central challenge in machine learning. Although numerous self-supervised learning methods have been proposed to address this challenge, a significant class of these approaches aligns the covariance or correlation matrix with the identity matrix. Despite impressive performance across various downstream tasks, these methods often suffer from biased sample risk, leading to substantial optimization shifts in mini-batch settings and complicating theoretical analysis. In this paper, we introduce a novel \underline{\bf Adv}ersarial \underline{\bf S}elf-\underline{\bf S}upervised Representation \underline{\bf L}earning (Adv-SSL) for unbiased transfer learning with no additional cost compared to its biased counterparts. Our approach not only outperforms the existing methods across multiple benchmark datasets but is also supported by comprehensive end-to-end theoretical guarantees. Our analysis reveals that the minimax optimization in Adv-SSL encourages representations to form well-separated clusters in the embedding space, provided there is sufficient upstream unlabeled data. As a result, our method achieves strong classification performance even with limited downstream labels, shedding new light on few-shot learning.