Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmark
This work addresses the challenge of noisy labels in real-world datasets for machine learning practitioners, but it is incremental as it synthesizes and benchmarks existing methods rather than introducing a novel approach.
The paper tackles the problem of detecting mislabeled examples in machine learning datasets by proposing a unified framework that views existing detection methods as probing trained models, and it benchmarks these methods on both artificial and realistic noise types to reveal new insights and limitations.
Mislabeled examples are ubiquitous in real-world machine learning datasets, advocating the development of techniques for automatic detection. We show that most mislabeled detection methods can be viewed as probing trained machine learning models using a few core principles. We formalize a modular framework that encompasses these methods, parameterized by only 4 building blocks, as well as a Python library that demonstrates that these principles can actually be implemented. The focus is on classifier-agnostic concepts, with an emphasis on adapting methods developed for deep learning models to non-deep classifiers for tabular data. We benchmark existing methods on (artificial) Completely At Random (NCAR) as well as (realistic) Not At Random (NNAR) labeling noise from a variety of tasks with imperfect labeling rules. This benchmark provides new insights as well as limitations of existing methods in this setup.