ReGLA: Efficient Receptive-Field Modeling with Gated Linear Attention Network
This addresses the problem of excessive latency in Transformer-based models for high-resolution visual applications, offering a state-of-the-art solution with incremental improvements in efficiency and performance.
The paper tackles the challenge of balancing accuracy and latency for lightweight models on high-resolution images by introducing ReGLA, a hybrid network combining efficient convolutions with gated linear attention, achieving 80.85% Top-1 accuracy on ImageNet-1K with 4.98 ms latency and outperforming comparable models on downstream tasks.
Balancing accuracy and latency on high-resolution images is a critical challenge for lightweight models, particularly for Transformer-based architectures that often suffer from excessive latency. To address this issue, we introduce \textbf{ReGLA}, a series of lightweight hybrid networks, which integrates efficient convolutions for local feature extraction with ReLU-based gated linear attention for global modeling. The design incorporates three key innovations: the Efficient Large Receptive Field (ELRF) module for enhancing convolutional efficiency while preserving a large receptive field; the ReLU Gated Modulated Attention (RGMA) module for maintaining linear complexity while enhancing local feature representation; and a multi-teacher distillation strategy to boost performance on downstream tasks. Extensive experiments validate the superiority of ReGLA; particularly the ReGLA-M achieves \textbf{80.85\%} Top-1 accuracy on ImageNet-1K at $224px$, with only \textbf{4.98 ms} latency at $512px$. Furthermore, ReGLA outperforms similarly scaled iFormer models in downstream tasks, achieving gains of \textbf{3.1\%} AP on COCO object detection and \textbf{3.6\%} mIoU on ADE20K semantic segmentation, establishing it as a state-of-the-art solution for high-resolution visual applications.