Tangma: A Tanh-Guided Activation Function with Learnable Parameters
This work addresses the need for more expressive and stable activation functions in deep learning, particularly for vision tasks, though it appears incremental as it builds on existing activation function designs.
The paper tackled the problem of improving activation functions in deep neural networks by introducing Tangma, a tanh-guided function with learnable parameters, which achieved the highest validation accuracy of 99.09% on MNIST and 78.15% on CIFAR-10, outperforming baselines like ReLU, Swish, and GELU.
Activation functions are key to effective backpropagation and expressiveness in deep neural networks. This work introduces Tangma, a new activation function that combines the smooth shape of the hyperbolic tangent with two learnable parameters: $α$, which shifts the curve's inflection point to adjust neuron activation, and $γ$, which adds linearity to preserve weak gradients and improve training stability. Tangma was evaluated on MNIST and CIFAR-10 using custom networks composed of convolutional and linear layers, and compared against ReLU, Swish, and GELU. On MNIST, Tangma achieved the highest validation accuracy of 99.09% and the lowest validation loss, demonstrating faster and more stable convergence than the baselines. On CIFAR-10, Tangma reached a top validation accuracy of 78.15%, outperforming all other activation functions while maintaining a competitive training loss. Tangma also showed improved training efficiency, with lower average epoch runtimes compared to Swish and GELU. These results suggest that Tangma performs well on standard vision tasks and enables reliable, efficient training. Its learnable design gives more control over activation behavior, which may benefit larger models in tasks such as image recognition or language modeling.