Injectivity of ReLU networks: perspectives from statistical physics
This work addresses a theoretical problem in understanding neural network invertibility for researchers in machine learning and statistical physics, but it is incremental as it builds on existing connections and leaves discrepancies unresolved.
The paper tackles the problem of determining when a single-layer ReLU neural network with random Gaussian weights is injective, meaning the input can be inferred from the output, by connecting it to the spherical perceptron spin glass model. Using replica symmetry-breaking theory and Gordon's min-max theorem, they derive analytical equations for the injectivity threshold that contradict a previous conjecture based on Euler characteristic methods.
When can the input of a ReLU neural network be inferred from its output? In other words, when is the network injective? We consider a single layer, $x \mapsto \mathrm{ReLU}(Wx)$, with a random Gaussian $m \times n$ matrix $W$, in a high-dimensional setting where $n, m \to \infty$. Recent work connects this problem to spherical integral geometry giving rise to a conjectured sharp injectivity threshold for $α= \frac{m}{n}$ by studying the expected Euler characteristic of a certain random set. We adopt a different perspective and show that injectivity is equivalent to a property of the ground state of the spherical perceptron, an important spin glass model in statistical physics. By leveraging the (non-rigorous) replica symmetry-breaking theory, we derive analytical equations for the threshold whose solution is at odds with that from the Euler characteristic. Furthermore, we use Gordon's min--max theorem to prove that a replica-symmetric upper bound refutes the Euler characteristic prediction. Along the way we aim to give a tutorial-style introduction to key ideas from statistical physics in an effort to make the exposition accessible to a broad audience. Our analysis establishes a connection between spin glasses and integral geometry but leaves open the problem of explaining the discrepancies.