Software-based Automatic Differentiation is Flawed
This highlights a critical issue for developers and users of machine learning and scientific computing tools, as it exposes potential inaccuracies in widely used differentiation methods.
The paper argues that software-based automatic differentiation frameworks, which rely on object-oriented programming to implement the chain rule, are flawed because they lack expression simplification mechanisms, leading to unbounded errors in results.
Various software efforts embrace the idea that object oriented programming enables a convenient implementation of the chain rule, facilitating so-called automatic differentiation via backpropagation. Such frameworks have no mechanism for simplifying the expressions (obtained via the chain rule) before evaluating them. As we illustrate below, the resulting errors tend to be unbounded.