Coarsening Optimization for Differentiable Programming
This addresses performance bottlenecks in differentiable programming for researchers and practitioners using AD in machine learning and scientific computing, representing a novel method rather than an incremental improvement.
This paper tackles the problem of runtime inefficiency in algorithmic differentiation (AD) by introducing coarsening optimization, which synergizes symbolic and algorithmic differentiation to reduce computations and allocations. Experiments on real-world applications show speedups ranging from several times to two orders of magnitude.
This paper presents a novel optimization for differentiable programming named coarsening optimization. It offers a systematic way to synergize symbolic differentiation and algorithmic differentiation (AD). Through it, the granularity of the computations differentiated by each step in AD can become much larger than a single operation, and hence lead to much reduced runtime computations and data allocations in AD. To circumvent the difficulties that control flow creates to symbolic differentiation in coarsening, this work introduces phi-calculus, a novel method to allow symbolic reasoning and differentiation of computations that involve branches and loops. It further avoids "expression swell" in symbolic differentiation and balance reuse and coarsening through the design of reuse-centric segment of interest identification. Experiments on a collection of real-world applications show that coarsening optimization is effective in speeding up AD, producing several times to two orders of magnitude speedups.