Memorization in Self-Supervised Learning Improves Downstream Generalization
This addresses the privacy and performance trade-off in SSL for machine learning practitioners, though it is incremental as it builds on existing memorization concepts.
The paper tackles the problem of defining and measuring memorization in self-supervised learning (SSL), where existing supervised definitions fail, and finds that significant memorization occurs despite regularization techniques, which is essential for achieving higher generalization performance on downstream tasks.
Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definitions of memorization from supervised learning rely on labels, they do not transfer to SSL. To address this gap, we propose SSLMem, a framework for defining memorization within SSL. Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not. Through comprehensive empirical analysis on diverse encoder architectures and datasets we highlight that even though SSL relies on large datasets and strong augmentations-both known in supervised learning as regularization techniques that reduce overfitting-still significant fractions of training data points experience high memorization. Through our empirical results, we show that this memorization is essential for encoders to achieve higher generalization performance on different downstream tasks.