Comparison of different convolutional neural network activation functions and methods for building ensembles
This work addresses improving CNN ensemble robustness for biomedical classification tasks, but it is incremental as it builds on existing ensemble methods with new activations.
The study compared CNN ensembles using different activation functions, including six new ones, and found that randomly replacing standard ReLU with varied activations in CNNs led to superior performance across fifteen biomedical datasets.
Recently, much attention has been devoted to finding highly efficient and powerful activation functions for CNN layers. Because activation functions inject different nonlinearities between layers that affect performance, varying them is one method for building robust ensembles of CNNs. The objective of this study is to examine the performance of CNN ensembles made with different activation functions, including six new ones presented here: 2D Mexican ReLU, TanELU, MeLU+GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The highest performing ensemble was built with CNNs having different activation layers that randomly replaced the standard ReLU. A comprehensive evaluation of the proposed approach was conducted across fifteen biomedical data sets representing various classification tasks. The proposed method was tested on two basic CNN architectures: Vgg16 and ResNet50. Results demonstrate the superiority in performance of this approach. The MATLAB source code for this study will be available at https://github.com/LorisNanni.