A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks
This addresses a bottleneck for researchers and practitioners using neural networks in scientific computing, such as physics-informed neural networks, by enabling efficient high-order derivative computations, though it is an incremental extension of existing methods.
The paper tackles the problem of exponentially increasing runtime for computing high-order derivatives in neural networks, which hinders applications like solving differential equations, and proposes n-TangentProp, an algorithm that computes exact derivatives in quasilinear time, validated with empirical demonstrations of faster training times in physics-informed neural networks.
The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose $n$-TangentProp, the natural extension of the TangentProp formalism \cite{simard1991tangent} to arbitrarily many derivatives. $n$-TangentProp computes the exact derivative $d^n/dx^n f(x)$ in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network $f$ with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where \ntp allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/n\_tangentprop.