Stochastic Multivariate Universal-Radix Finite-State Machine: a Theoretically and Practically Elegant Nonlinear Function Approximator
This addresses hardware efficiency for nonlinear function approximation in deep learning, offering incremental improvements in area and power consumption for specialized applications.
The paper tackles the hardware and compute overheads of nonlinear functions in deep neural networks by proposing a stochastic multivariate universal-radix finite-state machine (SMURF) that uses stochastic computing for efficient multivariate nonlinear function generation. Experiments show SMURF requires only 16.07% area and 14.45% power of Taylor-series approximation and 2.22% area of LUT schemes.
Nonlinearities are crucial for capturing complex input-output relationships especially in deep neural networks. However, nonlinear functions often incur various hardware and compute overheads. Meanwhile, stochastic computing (SC) has emerged as a promising approach to tackle this challenge by trading output precision for hardware simplicity. To this end, this paper proposes a first-of-its-kind stochastic multivariate universal-radix finite-state machine (SMURF) that harnesses SC for hardware-simplistic multivariate nonlinear function generation at high accuracy. We present the finite-state machine (FSM) architecture for SMURF, as well as analytical derivations of sampling gate coefficients for accurately approximating generic nonlinear functions. Experiments demonstrate the superiority of SMURF, requiring only 16.07% area and 14.45% power consumption of Taylor-series approximation, and merely 2.22% area of look-up table (LUT) schemes.