Task-Specific Activation Functions for Neuroevolution using Grammatical Evolution
This work addresses the need for more efficient neural networks, particularly for resource-constrained edge devices, by providing a method to enhance performance without added computational cost, though it is incremental as it builds on existing neuroevolution and activation function research.
The paper tackled the problem of suboptimal activation functions in neural networks by introducing Neuvo GEAF, which uses grammatical evolution to evolve task-specific activation functions, resulting in F1-score improvements of 2.4% to 9.4% over ReLU on binary classification datasets without increasing parameters.
Activation functions play a critical role in the performance and behaviour of neural networks, significantly impacting their ability to learn and generalise. Traditional activation functions, such as ReLU, sigmoid, and tanh, have been widely used with considerable success. However, these functions may not always provide optimal performance for all tasks and datasets. In this paper, we introduce Neuvo GEAF - an innovative approach leveraging grammatical evolution (GE) to automatically evolve novel activation functions tailored to specific neural network architectures and datasets. Experiments conducted on well-known binary classification datasets show statistically significant improvements in F1-score (between 2.4% and 9.4%) over ReLU using identical network architectures. Notably, these performance gains were achieved without increasing the network's parameter count, supporting the trend toward more efficient neural networks that can operate effectively on resource-constrained edge devices. This paper's findings suggest that evolved activation functions can provide significant performance improvements for compact networks while maintaining energy efficiency during both training and inference phases.