Identifying Training Stop Point with Noisy Labeled Data
This work is significant for researchers and practitioners training deep neural networks on real-world datasets where clean validation sets are often unavailable and noise ratios are unknown, offering a method to mitigate performance degradation due to noisy labels.
This paper addresses the problem of identifying an optimal training stop point for deep neural networks trained with noisy labels, a scenario where test accuracy initially rises and then falls. The authors propose a novel method, AutoTSP, that relies solely on training behavior and the entire training set to automatically determine this stop point, without requiring a clean validation set or prior knowledge of the noise ratio.
Training deep neural networks (DNNs) with noisy labels is a challenging problem due to over-parameterization. DNNs tend to essentially fit on clean samples at a higher rate in the initial stages, and later fit on the noisy samples at a relatively lower rate. Thus, with a noisy dataset, the test accuracy increases initially and drops in the later stages. To find an early stopping point at the maximum obtainable test accuracy (MOTA), recent studies assume either that i) a clean validation set is available or ii) the noise ratio is known, or, both. However, often a clean validation set is unavailable, and the noise estimation can be inaccurate. To overcome these issues, we provide a novel training solution, free of these conditions. We analyze the rate of change of the training accuracy for different noise ratios under different conditions to identify a training stop region. We further develop a heuristic algorithm based on a small-learning assumption to find a training stop point (TSP) at or close to MOTA. To the best of our knowledge, our method is the first to rely solely on the \textit{training behavior}, while utilizing the entire training set, to automatically find a TSP. We validated the robustness of our algorithm (AutoTSP) through several experiments on CIFAR-10, CIFAR-100, and a real-world noisy dataset for different noise ratios, noise types, and architectures.