A Simple and Efficient Stochastic Rounding Method for Training Neural Networks in Low Precision
This work addresses the challenge of efficient low-precision training for neural networks, offering incremental improvements in rounding methods.
The authors tackled the problem of training neural networks in low precision by introducing an improved stochastic rounding method, which achieved faster convergence and higher classification accuracy compared to conventional stochastic rounding and deterministic rounding, specifically with 16-bit fixed-point numbers.
Conventional stochastic rounding (CSR) is widely employed in the training of neural networks (NNs), showing promising training results even in low-precision computations. We introduce an improved stochastic rounding method, that is simple and efficient. The proposed method succeeds in training NNs with 16-bit fixed-point numbers and provides faster convergence and higher classification accuracy than both CSR and deterministic rounding-to-the-nearest method.