On the Geometry and Optimization of Polynomial Convolutional Networks
For theorists studying neural network expressivity and optimization landscapes, this provides rigorous geometric analysis of polynomial convolutional networks.
This paper studies convolutional neural networks with monomial activation functions, proving that their parameterization map is regular and an isomorphism almost everywhere up to rescaling. It computes the dimension and degree of the neuromanifold and derives an explicit formula for the number of critical points in regression optimization for generic large datasets.
We study convolutional neural networks with monomial activation functions. Specifically, we prove that their parameterization map is regular and is an isomorphism almost everywhere, up to rescaling the filters. By leveraging on tools from algebraic geometry, we explore the geometric properties of the image in function space of this map - typically referred to as neuromanifold. In particular, we compute the dimension and the degree of the neuromanifold, which measure the expressivity of the model, and describe its singularities. Moreover, for a generic large dataset, we derive an explicit formula that quantifies the number of critical points arising in the optimization of a regression loss.