Improving Deep Learning Optimization through Constrained Parameter Regularization
This addresses the issue of suboptimal regularization in deep learning for practitioners, though it is incremental as it builds on existing regularization methods.
The paper tackles the problem of uniform penalty in weight decay regularization by proposing Constrained Parameter Regularization (CPR), which enforces an upper bound on individual parameter matrices, resulting in improved performance over traditional weight decay in computer vision and language modeling tasks.
Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.