PENEX: AdaBoost-Inspired Neural Network Regularization
This work addresses the need for effective regularization in deep neural networks, offering an AdaBoost-inspired alternative that is incremental but with practical improvements.
The authors tackled the problem of improving neural network regularization by introducing PENEX, a new multi-class exponential loss formulation that is theoretically grounded and optimizable via first-order methods, showing it implicitly maximizes margins and exhibits a regularizing effect often better than established methods across computer vision and language tasks.
AdaBoost sequentially fits so-called weak learners to minimize an exponential loss, which penalizes mislabeled data points more severely than other loss functions like cross-entropy. Paradoxically, AdaBoost generalizes well in practice as the number of weak learners grows. In the present work, we introduce Penalized Exponential Loss (PENEX), a new formulation of the multi-class exponential loss that is theoretically grounded and, in contrast to the existing formulation, amenable to optimization via first-order methods. We demonstrate both empirically and theoretically that PENEX implicitly maximizes margins of data points. Also, we show that gradient increments on PENEX implicitly parameterize weak learners in the boosting framework. Across computer vision and language tasks, we show that PENEX exhibits a regularizing effect often better than established methods with similar computational cost. Our results highlight PENEX's potential as an AdaBoost-inspired alternative for effective training and fine-tuning of deep neural networks.