PolyLUT: Ultra-low Latency Polynomial Inference with Hardware-Aware Structured Pruning
This work addresses the problem of deploying efficient DNNs on FPGAs for applications requiring ultra-low latency, such as real-time detection systems, though it is incremental in optimizing existing FPGA-based methods.
The paper tackles the challenge of ultra-low latency DNN inference on FPGAs by proposing PolyLUT, which uses multivariate polynomials as building blocks and a hardware-aware structured pruning strategy to reduce LUT size and inputs per neuron, achieving significant latency and area improvements with comparable accuracy on tasks like network intrusion detection and MNIST.
Standard deep neural network inference involves the computation of interleaved linear maps and nonlinear activation functions. Prior work for ultra-low latency implementations has hardcoded these operations inside FPGA lookup tables (LUTs). However, FPGA LUTs can implement a much greater variety of functions. In this paper, we propose a novel approach to training DNNs for FPGA deployment using multivariate polynomials as the basic building block. Our method takes advantage of the flexibility offered by the soft logic, hiding the polynomial evaluation inside the LUTs with minimal overhead. By using polynomial building blocks, we achieve the same accuracy using considerably fewer layers of soft logic than by using linear functions, leading to significant latency and area improvements. LUT-based implementations also face a significant challenge: the LUT size grows exponentially with the number of inputs. Prior work relies on a priori fixed sparsity, with results heavily dependent on seed selection. To address this, we propose a structured pruning strategy using a bespoke hardware-aware group regularizer that encourages a particular sparsity pattern that leads to a small number of inputs per neuron. We demonstrate the effectiveness of PolyLUT on three tasks: network intrusion detection, jet identification at the CERN Large Hadron Collider, and MNIST.