Characterizing signal propagation to close the performance gap in unnormalized ResNets
This work addresses the practical challenges of batch normalization for researchers and practitioners in deep learning, offering a method to build high-performance image classifiers without normalization layers, though it is incremental as it builds on existing techniques.
The authors tackled the problem of eliminating batch normalization in ResNets, which causes practical issues like computational overhead and bugs, by developing analysis tools to characterize signal propagation and using Weight Standardization to maintain performance. Their networks achieved competitive results with state-of-the-art EfficientNets on ImageNet across various FLOP budgets.
Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs. Building on recent theoretical analyses of deep ResNets at initialization, we propose a simple set of analysis tools to characterize signal propagation on the forward pass, and leverage these tools to design highly performant ResNets without activation normalization layers. Crucial to our success is an adapted version of the recently proposed Weight Standardization. Our analysis tools show how this technique preserves the signal in networks with ReLU or Swish activation functions by ensuring that the per-channel activation means do not grow with depth. Across a range of FLOP budgets, our networks attain performance competitive with the state-of-the-art EfficientNets on ImageNet.