Convex Geometry of ReLU-layers, Injectivity on the Ball and Local Reconstruction
This work addresses theoretical challenges in understanding and inverting ReLU layers, which is incremental for neural network interpretability and robustness in machine learning.
The paper tackles the problem of verifying injectivity and reconstructing inputs for ReLU neural network layers on a closed ball, establishing a method to check injectivity under bias constraints and providing explicit reconstruction formulas.
The paper uses a frame-theoretic setting to study the injectivity of a ReLU-layer on the closed ball of $\mathbb{R}^n$ and its non-negative part. In particular, the interplay between the radius of the ball and the bias vector is emphasized. Together with a perspective from convex geometry, this leads to a computationally feasible method of verifying the injectivity of a ReLU-layer under reasonable restrictions in terms of an upper bound of the bias vector. Explicit reconstruction formulas are provided, inspired by the duality concept from frame theory. All this gives rise to the possibility of quantifying the invertibility of a ReLU-layer and a concrete reconstruction algorithm for any input vector on the ball.