Nish: A Novel Negative Stimulated Hybrid Activation Function
This work addresses the need for more efficient and robust activation functions in neural networks, but it appears incremental as it builds on existing functions like Mish.
The paper introduced the Nish activation function, which combines ReLU for positive inputs and a sinus-sigmoidal function for negative inputs, and reported that it achieves slightly better accuracy than Mish in classification tasks on popular benchmarks.
An activation function has a significant impact on the efficiency and robustness of the neural networks. As an alternative, we evolved a cutting-edge non-monotonic activation function, Negative Stimulated Hybrid Activation Function (Nish). It acts as a Rectified Linear Unit (ReLU) function for the positive region and a sinus-sigmoidal function for the negative region. In other words, it incorporates a sigmoid and a sine function and gaining new dynamics over classical ReLU. We analyzed the consistency of the Nish for different combinations of essential networks and most common activation functions using on several most popular benchmarks. From the experimental results, we reported that the accuracy rates achieved by the Nish is slightly better than compared to the Mish in classification.