Eigenvalue distribution of nonlinear models of random matrices
This work addresses theoretical foundations for understanding random neural networks, but it is incremental as it extends existing results to more general matrix distributions.
The paper tackles the problem of determining the asymptotic eigenvalue distribution of nonlinear random matrix ensembles, specifically for models resembling random neural networks with sub-Gaussian matrices and real analytic activation functions, extending prior Gaussian results and exploring multi-layer cases.
This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider $M= \frac{1}{m} YY^*$ with $Y=f(WX)$ where $W$ and $X$ are random rectangular matrices with i.i.d. centered entries. The function $f$ is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where $W$ and $X$ have sub-Gaussian tails and $f$ is real analytic. This extends a previous result where the case of Gaussian matrices $W$ and $X$ is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications.