Using accumulation to optimize deep residual neural nets
This work addresses a specific optimization problem for deep learning practitioners, but it is incremental as it builds directly on existing residual network techniques.
The paper tackled the problem of optimizing deep residual neural networks by proposing to include residuals from all lower layers, suitably normalized, to ensure equal contributions from previous layers. The result was an improvement over the original residual network method on the CIFAR-10 dataset.
Residual Neural Networks [1] won first place in all five main tracks of the ImageNet and COCO 2015 competitions. This kind of network involves the creation of pluggable modules such that the output contains a residual from the input. The residual in that paper is the identity function. We propose to include residuals from all lower layers, suitably normalized, to create the residual. This way, all previous layers contribute equally to the output of a layer. We show that our approach is an improvement on [1] for the CIFAR-10 dataset.