Differentiate Everything with a Reversible Embeded Domain-Specific Language
This addresses the inefficiency of checkpointing in general-purpose AD engines, which is a bottleneck for industrial applications requiring high-performance computing.
The paper tackles the high space overhead of reverse-mode automatic differentiation by proposing a reversible embedded domain-specific language that enables bidirectional program execution without implicit stack operations, achieving state-of-the-art performance in sparse matrix operations and machine learning applications.
Reverse-mode automatic differentiation (AD) suffers from the issue of having too much space overhead to trace back intermediate computational states for back-propagation. The traditional method to trace back states is called checkpointing that stores intermediate states into a global stack and restore state through either stack pop or re-computing. The overhead of stack manipulations and re-computing makes the general purposed (not tensor-based) AD engines unable to meet many industrial needs. Instead of checkpointing, we propose to use reverse computing to trace back states by designing and implementing a reversible programming eDSL, where a program can be executed bi-directionally without implicit stack operations. The absence of implicit stack operations makes the program compatible with existing compiler features, including utilizing existing optimization passes and compiling the code as GPU kernels. We implement AD for sparse matrix operations and some machine learning applications to show that our framework has the state-of-the-art performance.