Ensemble of Convolutional Neural Networks Trained with Different Activation Functions
This work addresses performance enhancement for biomedical image analysis using CNNs, but it is incremental as it builds on existing ensemble and activation function methods.
The authors tackled the problem of improving Convolutional Neural Network performance on small/medium biomedical datasets by proposing an ensemble of CNNs trained with different activation functions, including a novel one, and showed it outperforms standard ReLU-based CNNs with a p-value of 0.01 across over 10 datasets.
Activation functions play a vital role in the training of Convolutional Neural Networks. For this reason, to develop efficient and performing functions is a crucial problem in the deep learning community. Key to these approaches is to permit a reliable parameter learning, avoiding vanishing gradient problems. The goal of this work is to propose an ensemble of Convolutional Neural Networks trained using several different activation functions. Moreover, a novel activation function is here proposed for the first time. Our aim is to improve the performance of Convolutional Neural Networks in small/medium size biomedical datasets. Our results clearly show that the proposed ensemble outperforms Convolutional Neural Networks trained with standard ReLU as activation function. The proposed ensemble outperforms with a p-value of 0.01 each tested stand-alone activation function; for reliable performance comparison we have tested our approach in more than 10 datasets, using two well-known Convolutional Neural Network: Vgg16 and ResNet50. MATLAB code used here will be available at https://github.com/LorisNanni.