Exploration of Numerical Precision in Deep Neural Networks
This work addresses a computational cost issue for DNN practitioners, but it is incremental as it builds on known resilience observations without a general predictive result.
The paper tackled the problem of predicting DNN sensitivity to reduced numerical precision by emulating arbitrary bit-width with truncation and exploring model parameters, showing results on MNIST dataset with training accuracy impacts.
Reduced numerical precision is a common technique to reduce computational cost in many Deep Neural Networks (DNNs). While it has been observed that DNNs are resilient to small errors and noise, no general result exists that is capable of predicting a given DNN system architecture's sensitivity to reduced precision. In this project, we emulate arbitrary bit-width using a specified floating-point representation with a truncation method, which is applied to the neural network after each batch. We explore the impact of several model parameters on the network's training accuracy and show results on the MNIST dataset. We then present a preliminary theoretical investigation of the error scaling in both forward and backward propagations. We end with a discussion of the implications of these results as well as the potential for generalization to other network architectures.