A Non-monotonic Smooth Activation Function
This work addresses the need for better activation functions to enhance performance in tasks like classification and adversarial robustness for deep learning practitioners, though it appears incremental as it builds on existing activation function research.
The authors tackled the problem of improving activation functions in deep learning by proposing Sqish, a non-monotonic smooth function, which achieved an 8.21% improvement over ReLU in adversarial robustness and a 5.87% improvement in image classification on CIFAR100 with ShuffleNet V2.
Activation functions are crucial in deep learning models since they introduce non-linearity into the networks, allowing them to learn from errors and make adjustments, which is essential for learning complex patterns. The essential purpose of activation functions is to transform unprocessed input signals into significant output activations, promoting information transmission throughout the neural network. In this study, we propose a new activation function called Sqish, which is a non-monotonic and smooth function and an alternative to existing ones. We showed its superiority in classification, object detection, segmentation tasks, and adversarial robustness experiments. We got an 8.21% improvement over ReLU on the CIFAR100 dataset with the ShuffleNet V2 model in the FGSM adversarial attack. We also got a 5.87% improvement over ReLU on image classification on the CIFAR100 dataset with the ShuffleNet V2 model.