A Derivation of Feedforward Neural Network Gradients Using Fréchet Calculus
This work provides a theoretical improvement in gradient derivation for neural networks, but it is incremental as it refines existing methods without introducing new paradigms.
The authors tackled the problem of deriving gradients for feedforward neural networks by using Fréchet calculus, resulting in a more compact derivation and an efficient algorithm for gradient calculation that generalizes to architectures like convolutional networks.
We present a derivation of the gradients of feedforward neural networks using Fréchet calculus which is arguably more compact than the ones usually presented in the literature. We first derive the gradients for ordinary neural networks working on vectorial data and show how these derived formulas can be used to derive a simple and efficient algorithm for calculating a neural networks gradients. Subsequently we show how our analysis generalizes to more general neural network architectures including, but not limited to, convolutional networks.