The Riemannian Geometry associated to Gradient Flows of Linear Convolutional Networks
This provides a theoretical foundation for analyzing optimization in convolutional networks, which is incremental as it extends prior results from fully connected to convolutional architectures.
The paper tackles the problem of understanding gradient flows in linear convolutional networks by establishing that, unlike linear fully connected networks which require balanced initialization, the gradient flow on parameter space can be expressed as a Riemannian gradient flow on function space regardless of initialization for D-dimensional convolutions with D≥2, and for D=1 under certain stride conditions.
We study geometric properties of the gradient flow for learning deep linear convolutional networks. For linear fully connected networks, it has been shown recently that the corresponding gradient flow on parameter space can be written as a Riemannian gradient flow on function space (i.e., on the product of weight matrices) if the initialization satisfies a so-called balancedness condition. We establish that the gradient flow on parameter space for learning linear convolutional networks can be written as a Riemannian gradient flow on function space regardless of the initialization. This result holds for $D$-dimensional convolutions with $D \geq 2$, and for $D =1$ it holds if all so-called strides of the convolutions are greater than one. The corresponding Riemannian metric depends on the initialization.