Identity Matters in Deep Learning
This work addresses the challenge of training deep networks more effectively for researchers and practitioners in machine learning, though it builds incrementally on existing ideas like residual networks.
The paper tackled the problem of designing deep neural networks that can easily express identity transformations, showing that linear residual networks have no spurious local optima and ReLU residual networks have universal finite-sample expressivity. The result was a simple residual architecture that improved performance on CIFAR10, CIFAR100, and ImageNet benchmarks.
An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as \emph{batch normalization}, but was also key to the immense success of \emph{residual networks}. In this work, we put the principle of \emph{identity parameterization} on a more solid theoretical footing alongside further empirical progress. We first give a strikingly simple proof that arbitrarily deep linear residual networks have no spurious local optima. The same result for linear feed-forward networks in their standard parameterization is substantially more delicate. Second, we show that residual networks with ReLu activations have universal finite-sample expressivity in the sense that the network can represent any function of its sample provided that the model has more parameters than the sample size. Directly inspired by our theory, we experiment with a radically simple residual architecture consisting of only residual convolutional layers and ReLu activations, but no batch normalization, dropout, or max pool. Our model improves significantly on previous all-convolutional networks on the CIFAR10, CIFAR100, and ImageNet classification benchmarks.