DiracNets: Training Very Deep Neural Networks Without Skip-Connections
This work addresses the need for simpler and more efficient deep network architectures for image classification, though it is incremental as it builds directly on ResNet concepts.
The authors tackled the problem of training very deep neural networks without skip-connections by proposing a Dirac weight parameterization, achieving nearly the same performance as ResNet-1001 on CIFAR-10 with a 28-layer plain network and closely matching ResNets on ImageNet.
Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without explicit skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference, as both Dirac parameterization and batch normalization can be folded into convolutional filters, so that network becomes a simple chain of convolution-ReLU pairs. We are able to match ResNet-1001 accuracy on CIFAR-10 with 28-layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non-residual networks. The code and models for our experiments are available at https://github.com/szagoruyko/diracnets