Outlier-robust neural network training: variation regularization meets trimmed loss to prevent functional breakdown
This addresses the problem of outlier sensitivity in neural network training for predictive modeling, offering a robust solution that is incremental by combining existing concepts in a novel way.
The study tackled outlier-robust predictive modeling with neural networks by integrating transformed trimmed loss and higher-order variation regularization, resulting in a training procedure that retains a high functional breakdown point and ensures robustness to outlier contamination.
In this study, we tackle the challenge of outlier-robust predictive modeling using highly expressive neural networks. Our approach integrates two key components: (1) a transformed trimmed loss (TTL), a computationally efficient variant of the classical trimmed loss, and (2) higher-order variation regularization (HOVR), which imposes smoothness constraints on the prediction function. While traditional robust statistics typically assume low-complexity models such as linear and kernel models, applying TTL alone to modern neural networks may fail to ensure robustness, as their high expressive power allows them to fit both inliers and outliers, even when a robust loss is used. To address this, we revisit the traditional notion of breakdown point and adapt it to the nonlinear function setting, introducing a regularization scheme via HOVR that controls the model's capacity and suppresses overfitting to outliers. We theoretically establish that our training procedure retains a high functional breakdown point, thereby ensuring robustness to outlier contamination. We develop a stochastic optimization algorithm tailored to this framework and provide a theoretical guarantee of its convergence.