Most ReLU Networks Admit Identifiable Parameters
Provides foundational theoretical insight into parameter identifiability for ReLU networks, addressing a fundamental problem in understanding neural network representation.
This paper proves that for deep ReLU networks with input and hidden layers of width at least two, there exists an open set of parameters that are identifiable up to scaling and permutation, establishing that the functional dimension equals the number of parameters minus hidden neurons. It also shows a generic depth hierarchy where deeper networks cannot be represented by shallower ones.
We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.