Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models
This work provides incremental insights into activation function selection for deep learning practitioners, particularly in transfer learning scenarios.
The study investigated how ReLU and Leaky-ReLU activation functions affect deep learning model performance across architectures and data domains, finding that Leaky-ReLU is most effective with fewer trainable parameters and improves image classification in fully connected layers of pre-trained models like VGG-16.
The nonlinearity of activation functions used in deep learning models are crucial for the success of predictive models. There are several commonly used simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU (L-ReLU). In practice, these functions remarkably enhance the model accuracy. However, there is limited insight into the functionality of these nonlinear activation functions in terms of why certain models perform better than others. Here, we investigate the model performance when using ReLU or L-ReLU as activation functions in different model architectures and data domains. Interestingly, we found that the application of L-ReLU is mostly effective when the number of trainable parameters in a model is relatively small. Furthermore, we found that the image classification models seem to perform well with L-ReLU in fully connected layers, especially when pre-trained models such as the VGG-16 are used for the transfer learning.