Nonlinearity Enhanced Adaptive Activation Functions
This addresses the problem of enhancing neural network performance for machine learning practitioners, but it is incremental as it builds on existing activation functions.
The paper introduced a method to add parametric, learned nonlinearity to activation functions, improving neural network accuracy without major computational cost increases, as demonstrated with ReLU and other functions on MNIST and a CNN benchmark.
A general procedure for introducing parametric, learned, nonlinearity into activation functions is found to enhance the accuracy of representative neural networks without requiring significant additional computational resources. Examples are given based on the standard rectified linear unit (ReLU) as well as several other frequently employed activation functions. The associated accuracy improvement is quantified both in the context of the MNIST digit data set and a convolutional neural network (CNN) benchmark example.