Bayes-optimal Learning of Deep Random Networks of Extensive-width
This work provides theoretical insights into the optimal learning of deep random networks, which is incremental as it builds on existing asymptotic analyses in statistical learning theory.
The paper tackles the problem of learning a target function defined by a deep random neural network in the asymptotic limit of large samples, input dimension, and width, deriving a closed-form expression for the Bayes-optimal test error in regression and classification tasks. It finds that ridge regression and kernel regression achieve Bayes-optimal performance under certain conditions, while neural networks can achieve near-zero error with quadratically many samples when samples grow faster than dimension.
We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width are proportionally large. We propose a closed-form expression for the Bayes-optimal test error, for regression and classification tasks. We further compute closed-form expressions for the test errors of ridge regression, kernel and random features regression. We find, in particular, that optimally regularized ridge regression, as well as kernel regression, achieve Bayes-optimal performances, while the logistic loss yields a near-optimal test error for classification. We further show numerically that when the number of samples grows faster than the dimension, ridge and kernel methods become suboptimal, while neural networks achieve test error close to zero from quadratically many samples.