Deep Representation with ReLU Neural Networks
This work offers theoretical insight for the signal processing and machine learning communities, but it is incremental as it builds on existing frameworks without introducing a new paradigm.
The paper tackles the problem of understanding deep ReLU neural networks from a signal processing perspective, providing a precise description of affine linear representations and domain regions for input signals, with conditions suggested to stabilize learning independent of network depth.
We consider deep feedforward neural networks with rectified linear units from a signal processing perspective. In this view, such representations mark the transition from using a single (data-driven) linear representation to utilizing a large collection of affine linear representations tailored to particular regions of the signal space. This paper provides a precise description of the individual affine linear representations and corresponding domain regions that the (data-driven) neural network associates to each signal of the input space. In particular, we describe atomic decompositions of the representations and, based on estimating their Lipschitz regularity, suggest some conditions that can stabilize learning independent of the network depth. Such an analysis may promote further theoretical insight from both the signal processing and machine learning communities.