Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
This provides a more efficient tool for training neural networks by reducing computational demands compared to similar methods, though it is incremental in hyperparameter optimization.
The paper tackles hyperparameter selection by proposing a gradient-based method to adjust regularization hyperparameters during training, achieving optimal regularization levels on MNIST, SVHN, and CIFAR-10 with only 30% computational overhead.
Hyperparameter selection generally relies on running multiple full training trials, with selection based on validation set performance. We propose a gradient-based approach for locally adjusting hyperparameters during training of the model. Hyperparameters are adjusted so as to make the model parameter gradients, and hence updates, more advantageous for the validation cost. We explore the approach for tuning regularization hyperparameters and find that in experiments on MNIST, SVHN and CIFAR-10, the resulting regularization levels are within the optimal regions. The additional computational cost depends on how frequently the hyperparameters are trained, but the tested scheme adds only 30% computational overhead regardless of the model size. Since the method is significantly less computationally demanding compared to similar gradient-based approaches to hyperparameter optimization, and consistently finds good hyperparameter values, it can be a useful tool for training neural network models.