Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions
This work addresses theoretical gaps in understanding large-width properties of neural networks with heavy-tailed weights, which is incremental but relevant for researchers in machine learning theory and stability analysis.
The paper extends the characterization of infinitely wide limits for deep Stable neural networks from sub-linear to a broader class of activation functions, including linear and super-linear ones, using a generalized central limit theorem for heavy-tailed distributions. It finds that the scaling and stability of these limits depend on the activation function, highlighting a key difference from Gaussian neural networks.
There is a growing literature on the study of large-width properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed parameters or weights, and Gaussian stochastic processes. Motivated by some empirical and theoretical studies showing the potential of replacing Gaussian distributions with Stable distributions, namely distributions with heavy tails, in this paper we investigate large-width properties of deep Stable NNs, i.e. deep NNs with Stable-distributed parameters. For sub-linear activation functions, a recent work has characterized the infinitely wide limit of a suitable rescaled deep Stable NN in terms of a Stable stochastic process, both under the assumption of a ``joint growth" and under the assumption of a ``sequential growth" of the width over the NN's layers. Here, assuming a ``sequential growth" of the width, we extend such a characterization to a general class of activation functions, which includes sub-linear, asymptotically linear and super-linear functions. As a novelty with respect to previous works, our results rely on the use of a generalized central limit theorem for heavy tails distributions, which allows for an interesting unified treatment of infinitely wide limits for deep Stable NNs. Our study shows that the scaling of Stable NNs and the stability of their infinitely wide limits may depend on the choice of the activation function, bringing out a critical difference with respect to the Gaussian setting.