Algorithms and data structures for automatic precision estimation of neural networks
This work addresses reliability issues in neural network training and inference for practitioners, but it is incremental as it builds on existing library extensions.
The paper tackles the problem of computational inaccuracies accumulating in neural networks due to floating-point precision, showing that this affects inference, gradients, and model behavior, and proposes algorithms for automatic precision estimation to address it.
We describe algorithms and data structures to extend a neural network library with automatic precision estimation for floating point computations. We also discuss conditions to make estimations exact and preserve high computation performance of neural networks training and inference. Numerical experiments show the consequences of significant precision loss for particular values such as inference, gradients and deviations from mathematically predicted behavior. It turns out that almost any neural network accumulates computational inaccuracies. As a result, its behavior does not coincide with predicted by the mathematical model of neural network. This shows that tracking of computational inaccuracies is important for reliability of inference, training and interpretability of results.