On the Principles of ReLU Networks with One Hidden Layer
This work addresses the black-box nature of simple neural networks, which is a foundational issue for researchers in machine learning aiming to improve interpretability and control in training.
The paper tackles the problem of interpreting the solutions of two-layer ReLU networks trained via back-propagation, showing that for one-dimensional inputs, the training solution can be completely understood, and for higher-dimensional inputs, it can be well interpreted to some extent.
A neural network with one hidden layer or a two-layer network (regardless of the input layer) is the simplest feedforward neural network, whose mechanism may be the basis of more general network architectures. However, even to this type of simple architecture, it is also a ``black box''; that is, it remains unclear how to interpret the mechanism of its solutions obtained by the back-propagation algorithm and how to control the training process through a deterministic way. This paper systematically studies the first problem by constructing universal function-approximation solutions. It is shown that, both theoretically and experimentally, the training solution for the one-dimensional input could be completely understood, and that for a higher-dimensional input can also be well interpreted to some extent. Those results pave the way for thoroughly revealing the black box of two-layer ReLU networks and advance the understanding of deep ReLU networks.