ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction
This addresses the problem of computational inefficiency in dense vision tasks for researchers and practitioners, offering a novel method that is incremental but with strong specific gains.
The paper tackles efficient dense feature extraction in vision transformers by introducing ARTA, which uses adaptive mixed-resolution token allocation to focus computation on semantically complex areas, achieving state-of-the-art results on ADE20K and COCO-Stuff with substantially fewer FLOPs, such as 54.6 mIoU on ADE20K with ~100M parameters.
We present ARTA, a mixed-resolution coarse-to-fine vision transformer for efficient dense feature extraction. Unlike models that begin with dense high-resolution (fine) tokens, ARTA starts with low-resolution (coarse) tokens and uses a lightweight allocator to predict which regions require more fine tokens. The allocator iteratively predicts a semantic (class) boundary score and allocates additional tokens to patches above a low threshold, concentrating token density near boundaries while maintaining high sensitivity to weak boundary evidence. This targeted allocation encourages tokens to represent a single semantic class rather than a mixture of classes. Mixed-resolution attention enables interaction between coarse and fine tokens, focusing computation on semantically complex areas while avoiding redundant processing in homogeneous regions. Experiments demonstrate that ARTA achieves state-of-the-art results on ADE20K and COCO-Stuff with substantially fewer FLOPs, and delivers competitive performance on Cityscapes at markedly lower compute. For example, ARTA-Base attains 54.6 mIoU on ADE20K in the ~100M-parameter class while using fewer FLOPs and less memory than comparable backbones.