Constraining the outputs of ReLU neural networks
This work addresses foundational theoretical understanding of neural network expressivity for researchers in machine learning, but it is incremental as it builds on existing algebraic and geometric analyses.
The paper tackled the problem of characterizing the functions representable by ReLU neural networks by deriving polynomial equations from rank constraints on outputs across activation regions, providing insight into expressive and structural properties without reporting concrete numerical results.
We introduce a class of algebraic varieties naturally associated with ReLU neural networks, arising from the piecewise linear structure of their outputs across activation regions in input space, and the piecewise multilinear structure in parameter space. By analyzing the rank constraints on the network outputs within each activation region, we derive polynomial equations that characterize the functions representable by the network. We further investigate conditions under which these varieties attain their expected dimension, providing insight into the expressive and structural properties of ReLU networks.