TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks
This work addresses the need for more effective activation functions in neural networks, with potential applications in computer vision, natural language processing, and speech recognition, but it appears incremental as it builds on existing activation function concepts.
The paper tackles the problem of improving neural network performance by introducing the Truncated and Signed Square Root (TSSR) activation function, which is odd, nonlinear, monotone, and differentiable, and experiments show it outperforms other state-of-the-art activation functions.
Activation functions are essential components of neural networks. In this paper, we introduce a new activation function called the Truncated and Signed Square Root (TSSR) function. This function is distinctive because it is odd, nonlinear, monotone and differentiable. Its gradient is continuous and always positive. Thanks to these properties, it has the potential to improve the numerical stability of neural networks. Several experiments confirm that the proposed TSSR has better performance than other stat-of-the-art activation functions. The proposed function has significant implications for the development of neural network models and can be applied to a wide range of applications in fields such as computer vision, natural language processing, and speech recognition.