LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
This work addresses the problem of suboptimal regularization in deep learning for researchers and practitioners, offering an incremental improvement over existing label smoothing methods.
The authors tackled the limitation of conventional Label Smoothing (LS) by proposing LABO, a bi-level optimization framework that learns instance-specific label regularization, and demonstrated consistent improvements over conventional LS across seven machine translation and three image classification tasks.
Regularization techniques are crucial to improving the generalization performance and training efficiency of deep neural networks. Many deep learning algorithms rely on weight decay, dropout, batch/layer normalization to converge faster and generalize. Label Smoothing (LS) is another simple, versatile and efficient regularization which can be applied to various supervised classification tasks. Conventional LS, however, regardless of the training instance assumes that each non-target class is equally likely. In this work, we present a general framework for training with label regularization, which includes conventional LS but can also model instance-specific variants. Based on this formulation, we propose an efficient way of learning LAbel regularization by devising a Bi-level Optimization (LABO) problem. We derive a deterministic and interpretable solution of the inner loop as the optimal label smoothing without the need to store the parameters or the output of a trained model. Finally, we conduct extensive experiments and demonstrate our LABO consistently yields improvement over conventional label regularization on various fields, including seven machine translation and three image classification tasks across various