A Curriculum View of Robust Loss Functions
This addresses the issue of label noise robustness in machine learning, offering incremental improvements for practitioners using robust loss functions.
The paper tackles the problem of robust loss functions underfitting in the presence of label noise by analyzing training dynamics through a curriculum view, showing that simple adjustments can make these functions competitive with state-of-the-art methods.
Robust loss functions are designed to combat the adverse impacts of label noise, whose robustness is typically supported by theoretical bounds agnostic to the training dynamics. However, these bounds may fail to characterize the empirical performance as it remains unclear why robust loss functions can underfit. We show that most loss functions can be rewritten into a form with the same class-score margin and different sample-weighting functions. The resulting curriculum view provides a straightforward analysis of the training dynamics, which helps attribute underfitting to diminished average sample weights and noise robustness to larger weights for clean samples. We show that simple fixes to the curriculums can make underfitting robust loss functions competitive with the state-of-the-art, and training schedules can substantially affect the noise robustness even with robust loss functions. Code is available at \url{github}.