Activation degree thresholds and expressiveness of polynomial neural networks
This work provides theoretical insights into neural network expressiveness, which is foundational for machine learning researchers, though it is incremental as it builds on existing geometric frameworks.
The authors tackled the problem of understanding the expressive power of deep polynomial neural networks by introducing the activation degree threshold concept, proving its existence and a quadratic upper bound, and confirming a conjecture about high activation degrees.
We study the expressive power of deep polynomial neural networks through the geometry of their neurovariety. We introduce the notion of the activation degree threshold of a network architecture to express when the dimension of the neurovariety achieves its theoretical maximum. We prove the existence of the activation degree threshold for all polynomial neural networks without width-one bottlenecks and demonstrate a universal upper bound that is quadratic in the width of largest size. In doing so, we prove the high activation degree conjecture of Kileel, Trager, and Bruna. Certain structured architectures have exceptional activation degree thresholds, making them especially expressive in the sense of their neurovariety dimension. In this direction, we prove that polynomial neural networks with equi-width architectures are maximally expressive by showing their activation degree threshold is one.