Squareplus: A Softplus-Like Algebraic Rectifier
arXiv:2112.11687v131 citationsHas Code
Originality Synthesis-oriented
AI Analysis
This provides a more efficient alternative for resource-limited deep learning applications, though it is incremental as it builds on the softplus function.
The paper introduces squareplus, an activation function that approximates softplus using only algebraic operations, achieving ~6x faster evaluation on CPU and eliminating the need for transcendental functions.
We present squareplus, an activation function that resembles softplus, but which can be computed using only algebraic operations: addition, multiplication, and square-root. Because squareplus is ~6x faster to evaluate than softplus on a CPU and does not require access to transcendental functions, it may have practical value in resource-limited deep learning applications.