Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
This addresses the problem of reducing inference costs for deep learning practitioners by enabling more efficient hardware utilization, though it is incremental as it builds on existing quantization research.
The paper tackles the computational bottleneck of high-precision accumulators in deep neural networks by introducing a method to enable the use of cheaper 12-bit accumulators without significant accuracy loss, achieving this with no degradation in performance.
The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision of tensors visible by high-level frameworks (e.g., weights, activations, and gradients). However, current hardware still relies on high-accuracy core operations. Most significant is the operation of accumulating products. This high-precision accumulation operation is gradually becoming the main computational bottleneck. This is because, so far, the usage of low-precision accumulators led to a significant degradation in performance. In this work, we present a simple method to train and fine-tune high-end DNNs, to allow, for the first time, utilization of cheaper, $12$-bits accumulators, with no significant degradation in accuracy. Lastly, we show that as we decrease the accumulation precision further, using fine-grained gradient approximations can improve the DNN accuracy.