EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification
This work addresses overfitting in classification tasks, offering a novel ensemble approach that could benefit practitioners in machine learning, though it appears incremental in extending existing ensemble concepts to loss functions.
The authors tackled the problem of overfitting in classification by proposing EnsLoss, a method that combines multiple loss functions while preserving their convexity and calibration properties through stochastic ensembles. Experiments on 14 tabular and 46 image datasets showed improved effectiveness compared to fixed loss methods.
Empirical risk minimization (ERM) with a computationally feasible surrogate loss is a widely accepted approach for classification. Notably, the convexity and calibration (CC) properties of a loss function ensure consistency of ERM in maximizing accuracy, thereby offering a wide range of options for surrogate losses. In this article, we propose a novel ensemble method, namely EnsLoss, which extends the ensemble learning concept to combine loss functions within the ERM framework. A key feature of our method is the consideration on preserving the "legitimacy" of the combined losses, i.e., ensuring the CC properties. Specifically, we first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions and directly generating calibrated loss-derivatives. Therefore, inspired by Dropout, EnsLoss enables loss ensembles through one training process with doubly stochastic gradient descent (i.e., random batch samples and random calibrated loss-derivatives). We theoretically establish the statistical consistency of our approach and provide insights into its benefits. The numerical effectiveness of EnsLoss compared to fixed loss methods is demonstrated through experiments on a broad range of 14 OpenML tabular datasets and 46 image datasets with various deep learning architectures. Python repository and source code are available on GitHub at https://github.com/statmlben/ensloss.