Trajectory growth lower bounds for random sparse deep ReLU networks
This work addresses theoretical understanding of neural network expressivity for researchers, but it is incremental as it extends prior methods to more general distributions.
The paper tackles the problem of quantifying expressivity in deep ReLU networks by analyzing trajectory growth, generalizing existing results to provide simpler lower bounds for random networks with sparse weight distributions. It proves that exponential growth in depth persists even with sparsity, with the sparsity parameter affecting the exponent's base.
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected trajectory growth through random networks, for a more general class of weights distributions, including sparsely connected networks. We illustrate this approach by deriving bounds for sparse-Gaussian, sparse-uniform, and sparse-discrete-valued random nets. We prove that trajectory growth can remain exponential in depth with these new distributions, including their sparse variants, with the sparsity parameter appearing in the base of the exponent.