Data Leakage in Visual Datasets
This addresses a critical issue for computer vision researchers and practitioners by exposing widespread data leakage that undermines fair model assessment, though it is incremental in analyzing existing datasets.
The paper tackled the problem of data leakage in visual datasets, where images in evaluation benchmarks appear in training data, and found that all analyzed datasets exhibit some form of leakage, which compromises model evaluation reliability.
We analyze data leakage in visual datasets. Data leakage refers to images in evaluation benchmarks that have been seen during training, compromising fair model evaluation. Given that large-scale datasets are often sourced from the internet, where many computer vision benchmarks are publicly available, our efforts are focused into identifying and studying this phenomenon. We characterize visual leakage into different types according to its modality, coverage, and degree. By applying image retrieval techniques, we unequivocally show that all the analyzed datasets present some form of leakage, and that all types of leakage, from severe instances to more subtle cases, compromise the reliability of model evaluation in downstream tasks.