Spectrum concentration in deep residual learning: a free probability approach
This addresses the problem of slow training in deep residual networks for machine learning practitioners, offering a practical initialization method with significant speed gains.
The paper tackles the initialization of deep residual networks by introducing a free probability tool for non-Hermitian random matrices, enabling analysis of the Jacobian spectrum and proposing a rescaling scheme that leads to faster learning speeds, with empirical results showing orders of magnitude improvement.
We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully-connected deep ResNet for both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. In particular, we propose to rescale the classical random initialization by the number of residual units, so that the spectrum has the order of $O(1)$, when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.