Expressivity of Shallow Neural Networks Over Finite Fields
For researchers in neural network theory, this work provides a rigorous mathematical framework to understand how field characteristic affects expressivity, though it is incremental as it extends known algebraic geometry techniques to a specific architecture.
This paper studies the expressivity of shallow polynomial neural networks over finite fields by quantifying the cardinality of the neuromanifold, deriving lower and upper bounds via counting rational points, and showing a striking difference in neuromanifolds between characteristic zero and finite characteristic fields.
We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.