Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
This work addresses a fundamental issue in deep learning for researchers and practitioners, offering insights into gradient stability at initialization, though it is incremental by complementing existing mean field theory with rigorous finite-width corrections.
The paper tackles the problem of exploding and vanishing gradients in randomly initialized fully connected ReLU networks by providing a rigorous statistical analysis of gradient behavior, showing that the variance of gradient squares grows exponentially with an architecture-dependent constant beta, which is the sum of reciprocals of hidden layer widths.
We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, given by the sum of the reciprocals of the hidden layer widths. When beta is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.