ANAct: Adaptive Normalization for Activation Functions
This work addresses a specific bottleneck in neural network training by enhancing activation function normalization, offering incremental improvements for deep learning practitioners.
The paper tackles the negative effects of activation functions on gradient propagation in neural networks by proposing ANAct, an adaptive normalization method that maintains consistent gradient variance across layers, resulting in improved performance such as a 1.4% higher top-1 accuracy on ResNet50 with Tiny ImageNet compared to vanilla Swish.
In this paper, we investigate the negative effect of activation functions on forward and backward propagation and how to counteract this effect. First, We examine how activation functions affect the forward and backward propagation of neural networks and derive a general form for gradient variance that extends the previous work in this area. We try to use mini-batch statistics to dynamically update the normalization factor to ensure the normalization property throughout the training process, rather than only accounting for the state of the neural network after weight initialization. Second, we propose ANAct, a method that normalizes activation functions to maintain consistent gradient variance across layers and demonstrate its effectiveness through experiments. We observe that the convergence rate is roughly related to the normalization property. We compare ANAct with several common activation functions on CNNs and residual networks and show that ANAct consistently improves their performance. For instance, normalized Swish achieves 1.4\% higher top-1 accuracy than vanilla Swish on ResNet50 with the Tiny ImageNet dataset and more than 1.2\% higher with CIFAR-100.