Learning Deep Neural Networks under Agnostic Corrupted Supervision
This addresses the challenge of corrupted data in deep learning for practitioners, offering a unified solution for classification and regression, though it appears incremental as it builds on existing robust training approaches.
The paper tackles the problem of training deep neural networks with corrupted supervision by proposing an efficient robust algorithm that controls the collective impact of data points on gradients, achieving strong guarantees without assumptions on corruption type and showing limited impact on loss compared to state-of-the-art methods.
Training deep neural models in the presence of corrupted supervision is challenging as the corrupted data points may significantly impact the generalization performance. To alleviate this problem, we present an efficient robust algorithm that achieves strong guarantees without any assumption on the type of corruption and provides a unified framework for both classification and regression problems. Unlike many existing approaches that quantify the quality of the data points (e.g., based on their individual loss values), and filter them accordingly, the proposed algorithm focuses on controlling the collective impact of data points on the average gradient. Even when a corrupted data point failed to be excluded by our algorithm, the data point will have a very limited impact on the overall loss, as compared with state-of-the-art filtering methods based on loss values. Extensive experiments on multiple benchmark datasets have demonstrated the robustness of our algorithm under different types of corruption.