Statistical Challenges with Dataset Construction: Why You Will Never Have Enough Images
This highlights a critical problem for deploying AI models in safety-critical environments, as current evaluation methods are statistically unsound.
The paper argues that selecting representative image datasets for testing deep neural networks is often implausible, making performance statistics unreliable and failing to guarantee real-world performance, even with larger or bias-aware datasets.
Deep neural networks have achieved impressive performance on many computer vision benchmarks in recent years. However, can we be confident that impressive performance on benchmarks will translate to strong performance in real-world environments? Many environments in the real world are safety critical, and even slight model failures can be catastrophic. Therefore, it is crucial to test models rigorously before deployment. We argue, through both statistical theory and empirical evidence, that selecting representative image datasets for testing a model is likely implausible in many domains. Furthermore, performance statistics calculated with non-representative image datasets are highly unreliable. As a consequence, we cannot guarantee that models which perform well on withheld test images will also perform well in the real world. Creating larger and larger datasets will not help, and bias aware datasets cannot solve this problem either. Ultimately, there is little statistical foundation for evaluating models using withheld test sets. We recommend that future evaluation methodologies focus on assessing a model's decision-making process, rather than metrics such as accuracy.