Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space
This addresses the problem of inefficient hyperparameter tuning for deep learning practitioners, offering a method that is incremental but provides practical gains.
The paper tackled hyperparameter optimization in deep neural networks by sampling nonlocal paths instead of points, using parallel tempering to couple model instances and map hyperparameters to effective temperature. It resulted in faster training, improved resistance to overfitting, and a systematic decrease in absolute validation error, improving over benchmark results.
Hyperparameter optimization is both a practical issue and an interesting theoretical problem in training of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at a negligible additional computational cost, results can be improved by sampling nonlocal paths instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in training, which can be mapped to an effective temperature. The usually independent instances of the model are coupled and allowed to exchange their hyperparameters throughout the training using the well established parallel tempering technique of statistical physics. Each simulation corresponds then to a unique path, or history, in the joint hyperparameter/model-parameter space. We provide empirical tests of our method, in particular for dropout and learning rate optimization. We observed faster training and improved resistance to overfitting and showed a systematic decrease in the absolute validation error, improving over benchmark results.