Free Probability for predicting the performance of feed-forward fully connected neural networks
This work addresses the challenge of analyzing stability and performance in neural networks for researchers, though it appears incremental as it builds on prior free probability methods.
The authors tackled the problem of predicting neural network performance by developing a fast, reliable method to compute spectral densities of Jacobians using free probability theory, achieving up to 85% correlation with final test accuracies.
Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT). We present a reliable and very fast method for computing the associated spectral densities, for given architecture and initialization. This method has a controlled and proven convergence. Our technique is based on an homotopy method: it is an adaptative Newton-Raphson scheme which chains basins of attraction. In order to demonstrate the relevance of our method we show that the relevant FPT metrics computed before training are highly correlated to final test accuracies - up to 85\%. We also nuance the idea that learning happens at the edge of chaos by giving evidence that a very desirable feature for neural networks is the hyperbolicity of their Jacobian at initialization.