Learning with Bad Training Data via Iterative Trimmed Loss Minimization
It addresses the challenge of robust learning with corrupted data for machine learning practitioners, offering a generic solution applicable to various corruption types, though it builds on existing trimmed loss concepts.
The paper tackles the problem of learning model parameters when a fraction of training samples are corrupted, proposing an iterative trimmed loss minimization framework that recovers ground truth with linear convergence in generalized linear models and achieves state-of-the-art performance in settings like random label noise without requiring a-priori clean samples.
In this paper, we study a simple and generic framework to tackle the problem of learning model parameters when a fraction of the training samples are corrupted. We first make a simple observation: in a variety of such settings, the evolution of training accuracy (as a function of training epochs) is different for clean and bad samples. Based on this we propose to iteratively minimize the trimmed loss, by alternating between (a) selecting samples with lowest current loss, and (b) retraining a model on only these samples. We prove that this process recovers the ground truth (with linear convergence rate) in generalized linear models with standard statistical assumptions. Experimentally, we demonstrate its effectiveness in three settings: (a) deep image classifiers with errors only in labels, (b) generative adversarial networks with bad training images, and (c) deep image classifiers with adversarial (image, label) pairs (i.e., backdoor attacks). For the well-studied setting of random label noise, our algorithm achieves state-of-the-art performance without having access to any a-priori guaranteed clean samples.