Shifting Mean Activation Towards Zero with Bipolar Activation Functions
This addresses the need for simpler and more efficient training of deep networks, particularly in recurrent and convolutional architectures, though it is incremental as it builds on existing ReLU-family methods.
The authors tackled the problem of training deep neural networks without normalization layers by proposing bipolar activation functions that shift mean activations toward zero, achieving competitive results on language modeling tasks and faster training with lower test error on CIFAR-10.
We propose a simple extension to the ReLU-family of activation functions that allows them to shift the mean activation across a layer towards zero. Combined with proper weight initialization, this alleviates the need for normalization layers. We explore the training of deep vanilla recurrent neural networks (RNNs) with up to 144 layers, and show that bipolar activation functions help learning in this setting. On the Penn Treebank and Text8 language modeling tasks we obtain competitive results, improving on the best reported results for non-gated networks. In experiments with convolutional neural networks without batch normalization, we find that bipolar activations produce a faster drop in training error, and results in a lower test error on the CIFAR-10 classification task.