Algebraic Complexity and Neurovariety of Linear Convolutional Networks
This work addresses the theoretical understanding of optimization landscapes for researchers in algebraic complexity and neural network theory, but it is incremental as it builds on existing algebraic geometry tools to analyze specific network architectures.
The paper tackles the problem of understanding the algebraic complexity and critical points in training linear convolutional networks with one-dimensional filters and arbitrary strides, finding that the number of complex critical points equals the generic Euclidean distance degree of a Segre variety, which is significantly higher than in fully connected linear networks with the same parameter count.
In this paper, we study linear convolutional networks with one-dimensional filters and arbitrary strides. The neuromanifold of such a network is a semialgebraic set, represented by a space of polynomials admitting specific factorizations. Introducing a recursive algorithm, we generate polynomial equations whose common zero locus corresponds to the Zariski closure of the corresponding neuromanifold. Furthermore, we explore the algebraic complexity of training these networks employing tools from metric algebraic geometry. Our findings reveal that the number of all complex critical points in the optimization of such a network is equal to the generic Euclidean distance degree of a Segre variety. Notably, this count significantly surpasses the number of critical points encountered in the training of a fully connected linear network with the same number of parameters.