Learning on a Razor's Edge: the Singularity Bias of Polynomial Neural Networks
This work addresses the theoretical understanding of learning biases in neural networks for researchers in machine learning theory, though it is incremental as it builds on existing algebraic geometry approaches.
The authors theoretically analyzed the geometry of polynomial neural networks, showing that subnetworks correspond to singular subspaces in the function space, which often align with critical points in training dynamics, but found that this bias does not occur in convolutional networks.
Deep neural networks often infer sparse representations, converging to a subnetwork during the learning process. In this work, we theoretically analyze subnetworks and their bias through the lens of algebraic geometry. We consider fully-connected networks with polynomial activation functions, and focus on the geometry of the function space they parametrize, often referred to as neuromanifold. First, we compute the dimension of the subspace of the neuromanifold parametrized by subnetworks. Second, we show that this subspace is singular. Third, we argue that such singularities often correspond to critical points of the training dynamics. Lastly, we discuss convolutional networks, for which subnetworks and singularities are similarly related, but the bias does not arise.