If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
This work addresses the practical relevance of initialization strategies for neural network training in shallow-to-moderate depth settings, suggesting that efforts to optimize beyond critical initialization may be unnecessary.
The paper investigates whether critical initialization matters for ReLU networks when dropout limits trainable depth, finding through a large-scale analysis of over 12,000 networks that there is no significant performance difference across most non-critical initializations, with only extreme cases performing worse.
Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter? We conduct a large-scale controlled experiment, and perform a statistical analysis of over $12000$ trained networks. We find that (1) trainable networks show no statistically significant difference in performance over a wide range of non-critical initialisations; (2) for initialisations that show a statistically significant difference, the net effect on performance is small; (3) only extreme initialisations (very small or very large) perform worse than criticality. These findings also apply to standard ReLU networks of moderate depth as a special case of zero dropout. Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour.