Gradient-based Hyperparameter Optimization through Reversible Learning
This addresses the challenge of hyperparameter optimization for machine learning practitioners, offering a novel gradient-based approach rather than incremental improvements.
The paper tackles the problem of hyperparameter tuning by computing exact gradients of cross-validation performance with respect to hyperparameters, enabling optimization of thousands of hyperparameters such as step-size schedules and neural network architectures.
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.