MSE-Optimal Neural Network Initialization via Layer Fusion
This addresses initialization issues in deep learning for practitioners, but it is incremental as it builds on existing initialization and knowledge distillation methods.
The paper tackles the problem of neural network sensitivity to initialization by proposing FuseInit, a method to initialize shallower networks by fusing layers from deeper, randomly initialized networks, resulting in shallower networks performing better or as well as deeper counterparts.
Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. However, the use of stochastic gradient descent combined with the nonconvexity of the underlying optimization problems renders parameter learning susceptible to initialization. To address this issue, a variety of methods that rely on random parameter initialization or knowledge distillation have been proposed in the past. In this paper, we propose FuseInit, a novel method to initialize shallower networks by fusing neighboring layers of deeper networks that are trained with random initialization. We develop theoretical results and efficient algorithms for mean-square error (MSE)-optimal fusion of neighboring dense-dense, convolutional-dense, and convolutional-convolutional layers. We show experiments for a range of classification and regression datasets, which suggest that deeper neural networks are less sensitive to initialization and shallower networks can perform better (sometimes as well as their deeper counterparts) if initialized with FuseInit.