Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
This addresses a problem for researchers and practitioners in deep learning by providing incremental improvements to fixed-point training methods.
The paper tackled the instability in training deep convolutional networks with low numerical precision by theoretically connecting low precision to training stability and proposing methods to improve performance, achieving unspecified experimental verification.
It is known that training deep neural networks, in particular, deep convolutional networks, with aggressively reduced numerical precision is challenging. The stochastic gradient descent algorithm becomes unstable in the presence of noisy gradient updates resulting from arithmetic with limited numeric precision. One of the well-accepted solutions facilitating the training of low precision fixed point networks is stochastic rounding. However, to the best of our knowledge, the source of the instability in training neural networks with noisy gradient updates has not been well investigated. This work is an attempt to draw a theoretical connection between low numerical precision and training algorithm stability. In doing so, we will also propose and verify through experiments methods that are able to improve the training performance of deep convolutional networks in fixed point.