Gaussian Error Linear Units (GELUs)
This work addresses the need for better activation functions in neural networks for researchers and practitioners in AI, though it is incremental as it builds on existing functions like ReLU and ELU.
The authors tackled the problem of improving neural network activation functions by proposing the Gaussian Error Linear Unit (GELU), which weights inputs by their value rather than gating by sign like ReLUs, and found performance improvements across computer vision, natural language processing, and speech tasks.
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $xΦ(x)$, where $Φ(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.