Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form
This work provides incremental theoretical insights into neural network optimization, primarily for researchers in machine learning and linear algebra.
The paper tackles the linear algebra theory behind feedforward and recurrent neural networks, deriving new exact expressions for the Hessian matrix and showing that weight gradients and updates can be expressed as rank matrices based on time steps and batch size.
In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new exact expression for Hessian, which represents second order effects. We show that for $t$ time steps the weight gradient can be expressed as a rank-$t$ matrix, while the weight Hessian is as a sum of $t^{2}$ Kronecker products of rank-$1$ and $W^{T}AW$ matrices, for some matrix $A$ and weight matrix $W$. Also, we show that for a mini-batch of size $r$, the weight update can be expressed as a rank-$rt$ matrix. Finally, we briefly comment on the eigenvalues of the Hessian matrix.