A Non-Parametric Test to Detect Data-Copying in Generative Models
This addresses the challenge of overfitting detection for researchers and practitioners using generative models, though it appears incremental as it formalizes and tests an existing concept.
The paper tackles the problem of detecting data-copying, a form of overfitting in generative models where models memorize training data, by proposing a non-parametric test using three samples, and demonstrates its performance on canonical models and datasets.
Detecting overfitting in generative models is an important challenge in machine learning. In this work, we formalize a form of overfitting that we call {\em{data-copying}} -- where the generative model memorizes and outputs training samples or small variations thereof. We provide a three sample non-parametric test for detecting data-copying that uses the training set, a separate sample from the target distribution, and a generated sample from the model, and study the performance of our test on several canonical models and datasets. For code \& examples, visit https://github.com/casey-meehan/data-copying