ReCA: A Parametric ReLU Composite Activation Function
This work addresses the challenge of improving neural network performance for researchers and practitioners by introducing a new activation function, though it appears incremental as it builds upon ReLU.
The authors tackled the problem of identifying optimal activation functions for deep neural networks by proposing ReCA, a novel parametric activation function based on ReLU, which outperformed all baselines on state-of-the-art datasets across various complex architectures.
Activation functions have been shown to affect the performance of deep neural networks significantly. While the Rectified Linear Unit (ReLU) remains the dominant choice in practice, the optimal activation function for deep neural networks remains an open research question. In this paper, we propose a novel parametric activation function, ReCA, based on ReLU, which has been shown to outperform all baselines on state-of-the-art datasets using different complex neural network architectures.