Trainable Activation Function in Image Classification
This work addresses the limitation of fixed activation functions in deep learning, offering a domain-specific improvement for image classification tasks.
The paper tackles the problem of manually specified activation functions in neural networks by introducing trainable activation functions using Fourier series and linear combinations, achieving better performance than ReLU on the Cifar-10 dataset.
In the current research of neural networks, the activation function is manually specified by human and not able to change themselves during training. This paper focus on how to make the activation function trainable for deep neural networks. We use series and linear combination of different activation functions make activation functions continuously variable. Also, we test the performance of CNNs with Fourier series simulated activation(Fourier-CNN) and CNNs with linear combined activation function (LC-CNN) on Cifar-10 dataset. The result shows our trainable activation function reveals better performance than the most used ReLU activation function. Finally, we improves the performance of Fourier-CNN with Autoencoder, and test the performance of PSO algorithm in optimizing the parameters of networks