On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks
This work provides foundational insights into the topological expressiveness of ReLU networks, which is incremental but important for understanding neural network behavior in machine learning.
The paper investigates how the architecture of ReLU neural networks affects the geometry and topology of decision regions in binary classification, proving that for generic networks with a single hidden layer of dimension (n+1), decision regions can have at most one bounded connected component.
Let F:R^n -> R be a feedforward ReLU neural network. It is well-known that for any choice of parameters, F is continuous and piecewise (affine) linear. We lay some foundations for a systematic investigation of how the architecture of F impacts the geometry and topology of its possible decision regions for binary classification tasks. Following the classical progression for smooth functions in differential topology, we first define the notion of a generic, transversal ReLU neural network and show that almost all ReLU networks are generic and transversal. We then define a partially-oriented linear 1-complex in the domain of F and identify properties of this complex that yield an obstruction to the existence of bounded connected components of a decision region. We use this obstruction to prove that a decision region of a generic, transversal ReLU network F: R^n -> R with a single hidden layer of dimension (n + 1) can have no more than one bounded connected component.