On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats
This work addresses the challenge of training deep learning models efficiently with low-precision computations, which is crucial for resource-constrained environments, though it appears incremental as it extends existing SGD analysis to account for numerical errors.
The paper tackles the problem of analyzing Stochastic Gradient Descent (SGD) convergence when computations use low-precision number formats, which increase numerical error compared to single-precision, and presents deterministic and stochastic bounds to show the effect of the number format on convergence.
Deep learning models are dominating almost all artificial intelligence tasks such as vision, text, and speech processing. Stochastic Gradient Descent (SGD) is the main tool for training such models, where the computations are usually performed in single-precision floating-point number format. The convergence of single-precision SGD is normally aligned with the theoretical results of real numbers since they exhibit negligible error. However, the numerical error increases when the computations are performed in low-precision number formats. This provides compelling reasons to study the SGD convergence adapted for low-precision computations. We present both deterministic and stochastic analysis of the SGD algorithm, obtaining bounds that show the effect of number format. Such bounds can provide guidelines as to how SGD convergence is affected when constraints render the possibility of performing high-precision computations remote.