Fine-Grained Scene Image Classification with Modality-Agnostic Adapter
This addresses multi-modal feature fusion for fine-grained scene classification, offering an incremental improvement over existing methods.
The paper tackles fine-grained scene image classification by proposing a modality-gnostic adapter (MAA) that learns the importance of different modalities adaptively without prior architectural settings, achieving state-of-the-art results on benchmarks and allowing easy addition of new modalities to boost performance.
When dealing with the task of fine-grained scene image classification, most previous works lay much emphasis on global visual features when doing multi-modal feature fusion. In other words, models are deliberately designed based on prior intuitions about the importance of different modalities. In this paper, we present a new multi-modal feature fusion approach named MAA (Modality-Agnostic Adapter), trying to make the model learn the importance of different modalities in different cases adaptively, without giving a prior setting in the model architecture. More specifically, we eliminate the modal differences in distribution and then use a modality-agnostic Transformer encoder for a semantic-level feature fusion. Our experiments demonstrate that MAA achieves state-of-the-art results on benchmarks by applying the same modalities with previous methods. Besides, it is worth mentioning that new modalities can be easily added when using MAA and further boost the performance. Code is available at https://github.com/quniLcs/MAA.