A Closer Look at Double Backpropagation
This work addresses a theoretical gap for researchers and practitioners using neural networks with input derivatives, though it is incremental as it builds on existing double backpropagation methods.
The paper tackles the lack of a general description for derivatives in neural networks using double backpropagation, providing optimized backpropagation rules that reduce calculations for Frobenius-norm-penalties on Jacobians by roughly a third for locally linear activation functions.
In recent years, an increasing number of neural network models have included derivatives with respect to inputs in their loss functions, resulting in so-called double backpropagation for first-order optimization. However, so far no general description of the involved derivatives exists. Here, we cover a wide array of special cases in a very general Hilbert space framework, which allows us to provide optimized backpropagation rules for many real-world scenarios. This includes the reduction of calculations for Frobenius-norm-penalties on Jacobians by roughly a third for locally linear activation functions. Furthermore, we provide a description of the discontinuous loss surface of ReLU networks both in the inputs and the parameters and demonstrate why the discontinuities do not pose a big problem in reality.