SparsePixels: Efficient Convolution for Sparse Data on FPGAs
This work addresses latency reduction for FPGA-based inference in applications like neutrino detection, though it is incremental as it builds on existing sparse convolution methods.
The paper tackled the problem of high latency in convolutional neural network inference on FPGAs for sparse image data by introducing SparsePixels, a framework that selectively computes on active pixels, achieving a 73x speedup to 0.665 μs with minimal performance loss.
Inference of standard convolutional neural networks (CNNs) on FPGAs often incurs high latency and a long initiation interval due to the deep nested loops required to densely convolve every input pixel regardless of its feature value. However, input features can be spatially sparse in some image data, where semantic information may occupy only a small fraction of the pixels and most computation would be wasted on empty regions. In this work, we introduce SparsePixels, a framework that implements sparse convolution on FPGAs by selectively retaining and computing on a small subset of active pixels while ignoring the rest. We show that, for identifying neutrino interactions in naturally sparse LArTPC images with 4k pixels, a standard CNN with a compact size of 4k parameters incurs an inference latency of 48.665 $μ$s on an FPGA, whereas a sparse CNN of the same base architecture, computing on less than 1% of the input pixels, achieves a $\times 73$ speedup to 0.665 $μ$s with resource utilization well within on-chip budgets, trading only a small percent-level performance loss. This work aims to benefit future algorithm development for efficient data readout in modern experiments with latency requirements of microseconds or below.