Efficient Non-linear Calculators
This work addresses the need for efficient activation functions in ANN architectures, particularly for edge computing applications, though it appears incremental as it builds on existing non-linear function methods.
The paper tackles the problem of implementing efficient non-linear activation functions on digital hardware, presenting a novel algorithm that produces smooth quadratic non-linearities with symmetrical and asymmetrical variants, and results show these functions are highly amenable for multiplier-less implementations on ASICs or FPGAs, with thorough hardware resource usage investigations making a strong case for edge applications.
A novel algorithm for producing smooth nonlinearities on digital hardware is presented. The non-linearities are inherently quadratic and have both symmetrical and asymmetrical variants. The integer (and fixed point) implementation is highly amenable for use with digital gates on an ASIC or FPGA. The implementations are multiplier-less. Scaling of the non-linear output, as required in an LSTM cell, is integrated into the implementation. This too does not require a multiplier. The non-linearities are useful as activation functions in a variety of ANN architectures. The floating point mappings have been compared with other non-linearities and have been benchmarked. Results show that these functions should be considered in the ANN design phase. The hardware resource usage of the implementations have been thoroughly investigated. Our results make a strong case for implementions in edge applications. This document summarizes the findings and serves to give a quick overview of the outcomes of our research\footnote{The authors peer-reviewed manuscripts (available at https://doi.org/10.1016/j.neucom.2021.02.030) offer more detail and may be better suited for a thorough consideration}.