On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters
This addresses a foundational problem for machine learning practitioners by ensuring AD reliability in practical implementations, though it is incremental as it builds on prior theoretical work.
The paper tackles the correctness of automatic differentiation (AD) for neural networks with machine-representable parameters, proving that the incorrect set is empty for networks with bias parameters and providing a tight linear bound on the non-differentiable set, while showing AD always computes a Clarke subderivative.
Recent work has shown that forward- and reverse- mode automatic differentiation (AD) over the reals is almost always correct in a mathematically precise sense. However, actual programs work with machine-representable numbers (e.g., floating-point numbers), not reals. In this paper, we study the correctness of AD when the parameter space of a neural network consists solely of machine-representable numbers. In particular, we analyze two sets of parameters on which AD can be incorrect: the incorrect set on which the network is differentiable but AD does not compute its derivative, and the non-differentiable set on which the network is non-differentiable. For a neural network with bias parameters, we first prove that the incorrect set is always empty. We then prove a tight bound on the size of the non-differentiable set, which is linear in the number of non-differentiabilities in activation functions, and give a simple necessary and sufficient condition for a parameter to be in this set. We further prove that AD always computes a Clarke subderivative even on the non-differentiable set. We also extend these results to neural networks possibly without bias parameters.