Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
For practitioners of distributed deep learning, this work reduces the generalization gap of SignSGD relative to SGD, making sign-based compression more practical without sacrificing accuracy.
The paper improves SignSGD by deriving a small-batch convergence rate, adding annealed Gaussian noise as dithering, and adapting a hybrid switching strategy to SGD. On CIFAR-10, the calibrated switch achieves 92.18% test accuracy, outperforming pure SGD (91.38%) and pure SignSGD with momentum (90.82%).
SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.