High-Performance Large-Scale Image Recognition Without Normalization
This work addresses the need for stable, high-performance image recognition models without normalization, which is a significant improvement for computer vision applications, though it builds incrementally on prior efforts to remove normalization.
The paper tackled the problem of batch normalization's undesirable properties in image classification by developing an adaptive gradient clipping technique and Normalizer-Free ResNet models, achieving a new state-of-the-art top-1 accuracy of 86.5% on ImageNet and up to 8.7x faster training speeds compared to EfficientNet-B7.
Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets