Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation
This work addresses the high computational cost of semantic segmentation models for computer vision applications, offering an efficient solution with incremental improvements in efficiency.
The authors tackled the computational inefficiency of transformer-based semantic segmentation by proposing EDAFormer, which includes an Embedding-Free Transformer encoder and an Inference Spatial Reduction method, achieving state-of-the-art performance on benchmarks like ADE20K, Cityscapes, and COCO-Stuff while reducing computational cost by up to 61% on Cityscapes with minimal performance loss.
We present an Encoder-Decoder Attention Transformer, EDAFormer, which consists of the Embedding-Free Transformer (EFT) encoder and the all-attention decoder leveraging our Embedding-Free Attention (EFA) structure. The proposed EFA is a novel global context modeling mechanism that focuses on functioning the global non-linearity, not the specific roles of the query, key and value. For the decoder, we explore the optimized structure for considering the globality, which can improve the semantic segmentation performance. In addition, we propose a novel Inference Spatial Reduction (ISR) method for the computational efficiency. Different from the previous spatial reduction attention methods, our ISR method further reduces the key-value resolution at the inference phase, which can mitigate the computation-performance trade-off gap for the efficient semantic segmentation. Our EDAFormer shows the state-of-the-art performance with the efficient computation compared to the existing transformer-based semantic segmentation models in three public benchmarks, including ADE20K, Cityscapes and COCO-Stuff. Furthermore, our ISR method reduces the computational cost by up to 61% with minimal mIoU performance degradation on Cityscapes dataset. The code is available at https://github.com/hyunwoo137/EDAFormer.