A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks
This work addresses the problem of limited expressiveness in neural network activation functions for researchers and practitioners, though it appears incremental as it builds on existing activation function concepts.
The authors introduced the soft exponential activation function, which continuously interpolates between logarithmic, linear, and exponential functions, and demonstrated its potential to improve neural network learning by enabling exact calculations of operations like addition and multiplication.
We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable, and parameterized so that it can be trained as the rest of the network is trained. We hypothesize that soft exponential has the potential to improve neural network learning, as it can exactly calculate many natural operations that typical neural networks can only approximate, including addition, multiplication, inner product, distance, polynomials, and sinusoids.