Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network
This work addresses a specific bottleneck in CNNs by improving activation functions, offering incremental but practical gains for deep learning practitioners.
The paper tackles the problem of negative phase information flow in convolutional neural networks by proposing a novel nonlinear activation mechanism called reborn mechanism, which enhances model representation ability and achieves competitive or better performance with fewer parameters on various benchmark datasets.
This paper proposes a novel nonlinear activation mechanism typically for convolutional neural network (CNN), named as reborn mechanism. In sharp contrast to ReLU which cuts off the negative phase value, the reborn mechanism enjoys the capacity to reborn and reconstruct dead neurons. Compared to other improved ReLU functions, reborn mechanism introduces a more proper way to utilize the negative phase information. Extensive experiments validate that this activation mechanism is able to enhance the model representation ability more significantly and make the better use of the input data information while maintaining the advantages of the original ReLU function. Moreover, reborn mechanism enables a non-symmetry that is hardly achieved by traditional CNNs and can act as a channel compensation method, offering competitive or even better performance but with fewer learned parameters than traditional methods. Reborn mechanism was tested on various benchmark datasets, all obtaining better performance than previous nonlinear activation functions.