Deep Residual Networks and Weight Initialization
This work provides insights into why ResNets work well, which is incremental for researchers in deep learning.
The paper analyzed simplified ResNet models, arguing that ResNets' success is linked to their insensitivity to weight initialization, and demonstrated that batch normalization improves backpropagation in deep ResNets without tuning initial weights.
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.