Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation
This addresses optimization challenges in deep learning for researchers and practitioners, though it appears incremental as it builds on existing SGD frameworks.
The paper tackles the problem of inconsistent gradient magnitudes across layers in deep neural networks by introducing back-matching propagation, which is approximated to create a layer-wise adaptive learning rate strategy that outperforms standard SGD in experiments.
Stochastic gradient descent (SGD) has achieved great success in training deep neural network, where the gradient is computed through back-propagation. However, the back-propagated values of different layers vary dramatically. This inconsistence of gradient magnitude across different layers renders optimization of deep neural network with a single learning rate problematic. We introduce the back-matching propagation which computes the backward values on the layer's parameter and the input by matching backward values on the layer's output. This leads to solving a bunch of least-squares problems, which requires high computational cost. We then reduce the back-matching propagation with approximations and propose an algorithm that turns to be the regular SGD with a layer-wise adaptive learning rate strategy. This allows an easy implementation of our algorithm in current machine learning frameworks equipped with auto-differentiation. We apply our algorithm in training modern deep neural networks and achieve favorable results over SGD.