A Contrastive Learning Framework Empowered by Attention-based Feature Adaptation for Street-View Image Classification
This addresses the computational demands and limited fine-grained feature capture in street-view classification for applications like autonomous driving and urban analytics, representing an incremental improvement over existing CLIP adaptation methods.
The paper tackles the problem of street-view image attribute classification by proposing CLIP-MHAdapter, a lightweight adaptation method that adds a multi-head self-attention MLP to CLIP's patch tokens to capture fine-grained local features, achieving state-of-the-art or competitive accuracy on eight tasks with approximately 1.4 million trainable parameters.
Street-view image attribute classification is a vital downstream task of image classification, enabling applications such as autonomous driving, urban analytics, and high-definition map construction. It remains computationally demanding whether training from scratch, initialising from pre-trained weights, or fine-tuning large models. Although pre-trained vision-language models such as CLIP offer rich image representations, existing adaptation or fine-tuning methods often rely on their global image embeddings, limiting their ability to capture fine-grained, localised attributes essential in complex, cluttered street scenes. To address this, we propose CLIP-MHAdapter, a variant of the current lightweight CLIP adaptation paradigm that appends a bottleneck MLP equipped with multi-head self-attention operating on patch tokens to model inter-patch dependencies. With approximately 1.4 million trainable parameters, CLIP-MHAdapter achieves superior or competitive accuracy across eight attribute classification tasks on the Global StreetScapes dataset, attaining new state-of-the-art results while maintaining low computational cost. The code is available at https://github.com/SpaceTimeLab/CLIP-MHAdapter.