Measuring Déjà vu Memorization Efficiently
This work addresses the challenge of assessing memorization in large pre-trained models for researchers and practitioners, though it is incremental as it builds on existing déjà vu methods.
The paper tackled the problem of efficiently measuring data memorization in pre-trained open-source representation models, proposing simple methods to estimate dataset-level correlations without retraining, which enabled the first measurement of memorization in such models and found that open-source models typically have lower aggregate memorization than similar models trained on subsets of data.
Recent research has shown that representation learning models may accidentally memorize their training data. For example, the déjà vu method shows that for certain representation learning models and training images, it is sometimes possible to correctly predict the foreground label given only the representation of the background - better than through dataset-level correlations. However, their measurement method requires training two models - one to estimate dataset-level correlations and the other to estimate memorization. This multiple model setup becomes infeasible for large open-source models. In this work, we propose alternative simple methods to estimate dataset-level correlations, and show that these can be used to approximate an off-the-shelf model's memorization ability without any retraining. This enables, for the first time, the measurement of memorization in pre-trained open-source image representation and vision-language representation models. Our results show that different ways of measuring memorization yield very similar aggregate results. We also find that open-source models typically have lower aggregate memorization than similar models trained on a subset of the data. The code is available both for vision and vision language models.