Z-Error Loss for Training Neural Networks
This addresses training challenges for neural network practitioners dealing with noisy or outlier-prone datasets, offering a robust and adaptive solution.
The paper tackles the problem of outliers degrading neural network training by proposing the Z-Error Loss, which masks out-of-distribution data points in batches to minimize their influence, resulting in improved model performance and generalization.
Outliers introduce significant training challenges in neural networks by propagating erroneous gradients, which can degrade model performance and generalization. We propose the Z-Error Loss, a statistically principled approach that minimizes outlier influence during training by masking the contribution of data points identified as out-of-distribution within each batch. This method leverages batch-level statistics to automatically detect and exclude anomalous samples, allowing the model to focus its learning on the true underlying data structure. Our approach is robust, adaptive to data quality, and provides valuable diagnostics for data curation and cleaning.