Learning Rate Dropout
This addresses optimization inefficiencies in deep learning training, though it appears incremental as a modification to existing gradient descent methods.
The paper tackles the problem of deep neural networks converging slowly to poor local optima by proposing Learning Rate Dropout (LRD), a gradient descent technique that randomly sets some learning rates to zero to explore new descent paths, resulting in accelerated training and prevention of overfitting.
The performance of a deep neural network is highly dependent on its training, and finding better local optimal solutions is the goal of many optimization algorithms. However, existing optimization algorithms show a preference for descent paths that converge slowly and do not seek to avoid bad local optima. In this work, we propose Learning Rate Dropout (LRD), a simple gradient descent technique for training related to coordinate descent. LRD empirically aids the optimizer to actively explore in the parameter space by randomly setting some learning rates to zero; at each iteration, only parameters whose learning rate is not 0 are updated. As the learning rate of different parameters is dropped, the optimizer will sample a new loss descent path for the current update. The uncertainty of the descent path helps the model avoid saddle points and bad local minima. Experiments show that LRD is surprisingly effective in accelerating training while preventing overfitting.