On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation
This work addresses convergence issues in low-precision machine learning training, offering incremental improvements for efficient hardware implementations.
The study tackled the problem of gradient descent stagnation in low-precision floating-point computation by proposing two new stochastic rounding schemes that trade zero bias for a higher probability of preserving small gradients, proving they improve convergence rates for convex problems and validating this with experiments on logistic regression and neural networks using 8-bit formats.
When implementing the gradient descent method in low precision, the employment of stochastic rounding schemes helps to prevent stagnation of convergence caused by the vanishing gradient effect. Unbiased stochastic rounding yields zero bias by preserving small updates with probabilities proportional to their relative magnitudes. This study provides a theoretical explanation for the stagnation of the gradient descent method in low-precision computation. Additionally, we propose two new stochastic rounding schemes that trade the zero bias property with a larger probability to preserve small gradients. Our methods yield a constant rounding bias that, on average, lies in a descent direction. For convex problems, we prove that the proposed rounding methods typically have a beneficial effect on the convergence rate of gradient descent. We validate our theoretical analysis by comparing the performances of various rounding schemes when optimizing a multinomial logistic regression model and when training a simple neural network with an 8-bit floating-point format.