Fast Transformer Inference on ARM-Based HMPSoCs
For developers deploying transformers on ARM-based edge devices, this work provides a practical acceleration method with concrete speedups.
The paper addresses the challenge of deploying transformer models on resource-constrained ARM-based edge devices by implementing new transformer kernels in the ARM Compute Library, achieving up to 3x faster inference than state-of-the-art CPU/GPU implementations. Additionally, a cooperative CPU-GPU inference strategy reduces latency by up to 15.72% over single-processor execution.
Transformer models have set new performance standards for machine learning (ML) tasks. However, their resource-intensive deployment on resource-constrained edge devices for cloud-free, on-chip transformer inference remains challenging. The ARM Compute Library (ARM-CL) framework provides low-latency CNN inference on ARM-based edge devices but lacks support for transformer inference. In this work, we implement several new transformer kernels in ARM-CL to support native transformer execution. Our extended ARM-CL achieves up to three times faster transformer inference compared to state-of-the-art CPU/GPU implementations on an ARM-based embedded board. Furthermore, heterogeneous multi-processor system-on-chips (HMPSoCs) powering edge devices provide both embedded CPUs and GPUs. We introduce cooperative CPU-GPU transformer inference, which executes memory-intensive operations on the CPU while utilizing the GPU for highly parallelizable, compute-intensive operations. This cooperative execution, implemented with minimal overhead, further reduces transformer inference latency by up to 15.72% compared to the best single-processor inference on ARM-CL.