Symmetrical Gaussian Error Linear Units (SGELUs)
This work addresses performance and convergence issues in neural networks for researchers and practitioners, but it is incremental as it builds on existing activation functions like GELU and LiSHT.
The authors tackled the problem of improving neural network performance by proposing a new activation function, SGELU, which integrates stochastic regularization and symmetry to enable bidirectional convergence and avoid gradient diminishing, resulting in validated performance gains on MNIST classification and auto-encoder tasks.
In this paper, a novel neural network activation function, called Symmetrical Gaussian Error Linear Unit (SGELU), is proposed to obtain high performance. It is achieved by effectively integrating the property of the stochastic regularizer in the Gaussian Error Linear Unit (GELU) with the symmetrical characteristics. Combining with these two merits, the proposed unit introduces the capability of the bidirection convergence to successfully optimize the network without the gradient diminishing problem. The evaluations of SGELU against GELU and Linearly Scaled Hyperbolic Tangent (LiSHT) have been carried out on MNIST classification and MNIST auto-encoder, which provide great validations in terms of the performance, the convergence rate among these applications.