Any Deep ReLU Network is Shallow
This work addresses the interpretability of deep neural networks for researchers and practitioners by showing that complex deep models can be simplified into shallow, transparent forms.
The authors tackled the problem of understanding deep ReLU networks by proving that any such network can be rewritten as a functionally identical three-layer network with weights in the extended reals, and they provided an algorithm to compute the shallow network's explicit weights.
We constructively prove that every deep ReLU network can be rewritten as a functionally identical three-layer network with weights valued in the extended reals. Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. The resulting shallow network is transparent and used to generate explanations of the model s behaviour.