The Quest for the Golden Activation Function
This work addresses the need for automated optimization of activation functions in deep learning, offering a method to enhance performance on specific tasks, though it is incremental as it builds on existing genetic algorithm ideas.
The paper tackles the problem of manually selecting activation functions in deep neural networks by using genetic algorithms to learn task-specific activation functions, demonstrating improved results on three image classification benchmarks.
Deep Neural Networks have been shown to be beneficial for a variety of tasks, in particular allowing for end-to-end learning and reducing the requirement for manual design decisions. However, still many parameters have to be chosen in advance, also raising the need to optimize them. One important, but often ignored system parameter is the selection of a proper activation function. Thus, in this paper we target to demonstrate the importance of activation functions in general and show that for different tasks different activation functions might be meaningful. To avoid the manual design or selection of activation functions, we build on the idea of genetic algorithms to learn the best activation function for a given task. In addition, we introduce two new activation functions, ELiSH and HardELiSH, which can easily be incorporated in our framework. In this way, we demonstrate for three different image classification benchmarks that different activation functions are learned, also showing improved results compared to typically used baselines.