On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case
This work provides a more general theoretical understanding of random matrix properties in deep neural networks for researchers working on the mathematical foundations of deep learning, extending previous results to a broader class of input distributions.
This paper generalizes previous results on the singular value distribution of random matrices in deep neural networks. It extends the analysis from Gaussian data matrices to independent and identically distributed random variables with zero mean and finite fourth moment, thereby extending macroscopic universality.
We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the population covariance matrices assumed to be non-random or random but independent of the random data matrix in statistics and random matrix theory are now certain functions of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [25, 13] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability has to be justified. The justification has been given in [22] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper we use another, more streamlined, version of the techniques of random matrix theory to generalize the results of [22] to the case where the entries of the synaptic weight matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, extends the property of the so-called macroscopic universality on the considered random matrices.