WRPN: Training and Inference using Wide Reduced-Precision Networks
This addresses the need for efficient deep learning models in computer vision by enabling reduced-precision training and inference without accuracy loss, though it is incremental as it builds on prior quantization work.
The paper tackles the problem of reduced-precision activations hurting accuracy in deep neural networks by proposing WRPN, a scheme that uses quantization and increased filter maps to maintain or surpass baseline accuracy, achieving better results on ILSVRC-12 with reduced computational costs.
For computer vision applications, prior works have shown the efficacy of reducing the numeric precision of model parameters (network weights) in deep neural networks but also that reducing the precision of activations hurts model accuracy much more than reducing the precision of model parameters. We study schemes to train networks from scratch using reduced-precision activations without hurting the model accuracy. We reduce the precision of activation maps (along with model parameters) using a novel quantization scheme and increase the number of filter maps in a layer, and find that this scheme compensates or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly reduce the dynamic memory footprint, memory bandwidth, computational energy and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results using our proposed schemes and show that our results are better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.