Exploring vision transformer layer choosing for semantic segmentation
This work addresses the need for dynamic feature selection in semantic segmentation, offering a plug-in module for improved performance, though it is incremental as it builds on existing Vision Transformer frameworks.
The paper tackles the problem of manually selecting fixed layers in Vision Transformers for semantic segmentation by proposing ViTController, a neck network for adaptive fusion and feature selection, which surpasses previous state-of-the-art methods on different datasets and models.
Extensive work has demonstrated the effectiveness of Vision Transformers. The plain Vision Transformer tends to obtain multi-scale features by selecting fixed layers, or the last layer of features aiming to achieve higher performance in dense prediction tasks. However, this selection is often based on manual operation. And different samples often exhibit different features at different layers (e.g., edge, structure, texture, detail, etc.). This requires us to seek a dynamic adaptive fusion method to filter different layer features. In this paper, unlike previous encoder and decoder work, we design a neck network for adaptive fusion and feature selection, called ViTController. We validate the effectiveness of our method on different datasets and models and surpass previous state-of-the-art methods. Finally, our method can also be used as a plug-in module and inserted into different networks.