The Geometric Structure of Fully-Connected ReLU Layers
This provides theoretical insights into the geometry of ReLU networks, which is incremental as it extends prior work on convolutional networks to fully-connected layers.
The authors formalized the geometric structure of fully-connected ReLU layers, showing they partition the input domain into sectors where the layer simplifies to a projection onto a polyhedral cone followed by an affine transformation, and proved that a single hidden ReLU-layer network can generate only d different decision boundaries modulo an affine transformation.
We formalize and interpret the geometric structure of $d$-dimensional fully connected ReLU layers in neural networks. The parameters of a ReLU layer induce a natural partition of the input domain, such that the ReLU layer can be significantly simplified in each sector of the partition. This leads to a geometric interpretation of a ReLU layer as a projection onto a polyhedral cone followed by an affine transformation, in line with the description in [doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU activations. Further, this structure facilitates simplified expressions for preimages of the intersection between partition sectors and hyperplanes, which is useful when describing decision boundaries in a classification setting. We investigate this in detail for a feed-forward network with one hidden ReLU-layer, where we provide results on the geometric complexity of the decision boundary generated by such networks, as well as proving that modulo an affine transformation, such a network can only generate $d$ different decision boundaries. Finally, the effect of adding more layers to the network is discussed.