SignReLU neural network and its approximation ability
This work addresses the need for better activation functions in deep learning, offering an incremental improvement for researchers and practitioners in neural network design.
The paper tackles the problem of improving neural network approximation by introducing the SignReLU activation function, demonstrating theoretically that it outperforms rational and ReLU networks and showing competitive practical performance in numerical experiments.
Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU.