FuSeConv: Fully Separable Convolutions for Fast Inference on Systolic Arrays
This work addresses the problem of fast DNN inference on edge devices for applications requiring low latency and high efficiency, representing an incremental improvement in hardware-aware neural network design.
The paper tackles the inefficiency of combining depthwise separable convolutions with systolic arrays by proposing FuSeConv, a fully separable convolution method that achieves 3x-7x speed-up on MobileNet networks with comparable accuracy on ImageNet.
Both efficient neural networks and hardware accelerators are being explored to speed up DNN inference on edge devices. For example, MobileNet uses depthwise separable convolution to achieve much lower latency, while systolic arrays provide much higher performance per watt. Interestingly however, the combination of these two ideas is inefficient: The computational patterns of depth-wise separable convolution are not systolic and lack data reuse to saturate the systolic array's constrained dataflow. In this paper, we propose FuSeConv (Fully-Separable Convolution) as a drop-in replacement for depth-wise separable convolution. FuSeConv generalizes the decomposition of convolutions fully to separable 1D convolutions along spatial and depth dimensions. The resultant computation is systolic and efficiently utilizes the systolic array with a slightly modified dataflow. With FuSeConv, we achieve a significant speed-up of 3x-7x with the MobileNet family of networks on a systolic array of size 64x64, with comparable accuracy on the ImageNet dataset. The high speed-up motivates exploration of hardware-aware Neural Operator Search (NOS) in complement to ongoing efforts on Neural Architecture Search (NAS).