On the Convergence of A Family of Robust Losses for Stochastic Gradient Descent
This addresses noisy labels in applications like crowdsourcing, but it is incremental as it builds on existing SGD methods with new loss functions.
The paper tackles the problem of noisy labels in Stochastic Gradient Descent (SGD) by introducing a family of robust losses, achieving a convergence rate of O(1/T) and showing improved robustness over baselines on six real-world datasets.
The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal that the convergence rate is O(1/T) for SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.