A Memory-Efficient Framework for Deformable Transformer with Neural Architecture Search
This work addresses memory efficiency challenges for deploying deformable transformers on edge hardware, representing an incremental improvement over prior acceleration methods.
The paper tackled the irregular memory access patterns in Deformable Attention Transformers, which hinder hardware deployment, by proposing a hardware-friendly optimization framework using neural architecture search and a slicing strategy, resulting in only a 0.2% accuracy drop on ImageNet-1K and reducing DRAM access times to 18% compared to existing methods on FPGA.
Deformable Attention Transformers (DAT) have shown remarkable performance in computer vision tasks by adaptively focusing on informative image regions. However, their data-dependent sampling mechanism introduces irregular memory access patterns, posing significant challenges for efficient hardware deployment. Existing acceleration methods either incur high hardware overhead or compromise model accuracy. To address these issues, this paper proposes a hardware-friendly optimization framework for DAT. First, a neural architecture search (NAS)-based method with a new slicing strategy is proposed to automatically divide the input feature into uniform patches during the inference process, avoiding memory conflicts without modifying model architecture. The method explores the optimal slice configuration by jointly optimizing hardware cost and inference accuracy. Secondly, an FPGA-based verification system is designed to test the performance of this framework on edge-side hardware. Algorithm experiments on the ImageNet-1K dataset demonstrate that our hardware-friendly framework can maintain have only 0.2% accuracy drop compared to the baseline DAT. Hardware experiments on Xilinx FPGA show the proposed method reduces DRAM access times to 18% compared with existing DAT acceleration methods.