Understanding Automatic Differentiation Pitfalls
It addresses a practical problem for users of AD tools by clarifying common errors, though it is incremental as it builds on existing knowledge without introducing new methods.
The paper categorizes systematic pitfalls in automatic differentiation (AD) where computed derivatives can be misinterpreted as incorrect, using examples like chaos and fixed-point loops, and reviews debugging techniques to help users avoid and detect these issues.
Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives computed by AD could be interpreted as incorrect. These pitfalls occur systematically across tools and approaches. In this paper we broadly categorize problematic usages of AD and illustrate each category with examples such as chaos, time-averaged oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers. We also review debugging techniques and their effectiveness in these situations. With this article we hope to help readers avoid unexpected behavior, detect problems more easily when they occur, and have more realistic expectations from AD tools.