FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Clues
This work addresses the problem of improving multi-modal reasoning for AI systems, but it appears incremental as it builds on existing pre-trained models with novel modules.
The paper tackled multi-modal reasoning by proposing the FSMR model, which uses feature swapping and cross-attention to enhance understanding between text and images, achieving superior performance over state-of-the-art baselines on the PMR dataset.
Multi-modal reasoning plays a vital role in bridging the gap between textual and visual information, enabling a deeper understanding of the context. This paper presents the Feature Swapping Multi-modal Reasoning (FSMR) model, designed to enhance multi-modal reasoning through feature swapping. FSMR leverages a pre-trained visual-language model as an encoder, accommodating both text and image inputs for effective feature representation from both modalities. It introduces a unique feature swapping module, enabling the exchange of features between identified objects in images and corresponding vocabulary words in text, thereby enhancing the model's comprehension of the interplay between images and text. To further bolster its multi-modal alignment capabilities, FSMR incorporates a multi-modal cross-attention mechanism, facilitating the joint modeling of textual and visual information. During training, we employ image-text matching and cross-entropy losses to ensure semantic consistency between visual and language elements. Extensive experiments on the PMR dataset demonstrate FSMR's superiority over state-of-the-art baseline models across various performance metrics.