Stochastic Rounding Increases Small Singular Values
For researchers using low-precision arithmetic in numerical analysis and machine learning, this work provides a more general characterization of SR's spectral regularization effects, extending previous results to broader matrix aspect ratios and spectrum regions.
This paper shows that stochastic rounding (SR) regularizes singular values not only for extremely tall-and-thin matrices but also for matrices with constant aspect ratio, and that it lifts entire clusters of tail singular values rather than just the smallest one.
Over the past half-dozen years, stochastic rounding (SR) has regained significant attention as a quantization scheme for low-precision floating-point arithmetic, with applications spanning numerical analysis and modern machine learning systems. Recent work has shown that SR acts as an implicit regularizer by increasing the smallest singular value of extremely tall-and-thin (or, symmetrically, short-and-fat) matrices. In this work, we substantially sharpen and extend this understanding in two directions. First, we show that the regularization effect of SR is not restricted to extreme aspect ratio regimes: it persists for matrices with constant aspect ratio. Second, we demonstrate that SR does not merely regularize the smallest singular value, but instead lifts entire clusters of singular values at the tail of the spectrum. Together, these results provide a more general characterization of stochastic rounding as a spectral regularizer, revealing that its effects extend beyond extremal aspect ratios and act on a broader portion of the singular value spectrum.