A Concept-Centric Approach to Multi-Modality Learning
This addresses multi-modality AI efficiency for researchers, though it appears incremental as it matches rather than surpasses benchmarks.
The authors tackled multi-modality learning by introducing a framework with a modality-agnostic concept space and modality-specific projection models, achieving performance comparable to benchmarks with more efficient learning curves.
In an effort to create a more efficient AI system, we introduce a new multi-modality learning framework that leverages a modality-agnostic concept space possessing abstract knowledge and a set of modality-specific projection models tailored to process distinct modality inputs and map them onto the concept space. Decoupled from specific modalities and their associated projection models, the concept space focuses on learning abstract knowledge that is universally applicable across modalities. Subsequently, the knowledge embedded into the concept space streamlines the learning processes of modality-specific projection models. We evaluate our framework on two popular tasks: Image-Text Matching and Visual Question Answering. Our framework achieves performance on par with benchmark models while demonstrating more efficient learning curves.