On Stochastic Rounding with Few Random Bits
This work addresses a practical issue for developers using low-precision arithmetic in large-scale computations, but it is incremental as it focuses on optimizing an existing technique rather than introducing a new paradigm.
The paper tackles the problem of reducing the number of random bits needed for stochastic rounding in low-precision computations, showing that some implementations introduce significant bias not present in infinite-bit cases. It demonstrates the impact of these biases in machine learning examples, highlighting a new class of configuration parameters for practitioners.
Large-scale numerical computations make increasing use of low-precision (LP) floating point formats and mixed precision arithmetic, which can be enhanced by the technique of stochastic rounding (SR), that is, rounding an intermediate high-precision value up or down randomly as a function of the value's distance to the two rounding candidates. Stochastic rounding requires, in addition to the high-precision input value, a source of random bits. As the provision of high-quality random bits is an additional computational cost, it is of interest to require as few bits as possible while maintaining the desirable properties of SR in a given computation, or computational domain. This paper examines a number of possible implementations of few-bit stochastic rounding (FBSR), and shows how several natural implementations can introduce sometimes significant bias into the rounding process, which are not present in the case of infinite-bit, infinite-precision examinations of these implementations. The paper explores the impact of these biases in machine learning examples, and hence opens another class of configuration parameters of which practitioners should be aware when developing or adopting low-precision floating point. Code is available at http://github.com/graphcore-research/arith25-stochastic-rounding.