On the Equivalence of Automatic and Symbolic Differentiation
This clarifies foundational concepts in machine learning and computational mathematics, correcting a common misunderstanding in the field.
The paper tackles the misconception that reverse mode automatic differentiation and symbolic differentiation are fundamentally different, showing they perform the same operations when computing derivatives and refuting the claim that symbolic differentiation uniquely suffers from 'expression swell'.
We show that reverse mode automatic differentiation and symbolic differentiation are equivalent in the sense that they both perform the same operations when computing derivatives. This is in stark contrast to the common claim that they are substantially different. The difference is often illustrated by claiming that symbolic differentiation suffers from "expression swell" whereas automatic differentiation does not. Here, we show that this statement is not true. "Expression swell" refers to the phenomenon of a much larger representation of the derivative as opposed to the representation of the original function.