GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention
For researchers in multimodal learning, this work addresses the limitation of existing attention mechanisms that fail to explicitly model joint dependencies across multiple modalities, offering a more expressive and efficient alternative.
The paper introduces Volumetric Multimodal cross-Attention (VMA), a new attention mechanism that models joint interactions across multiple modalities by computing the volume spanned by query and key vectors, enabling any-order modality interactions. The proposed GRAMformer architecture integrates VMA and shows improved effectiveness and efficiency on multimodal learning tasks.
Transformer-based multimodal models rely on attention mechanisms to integrate information across heterogeneous modalities. Despite their success, existing multimodal attention formulations compute their scores through collections of pairwise dot-product interactions or by concatenating all the modalities into the keys, even when multiple modalities should be jointly involved. As a consequence, current approaches either incur quadratic complexity in the number of modalities or fail to explicitly model interactions that depend on the joint configuration of multiple representations. In this work, we introduce the Volumetric Multimodal cross-Attention (VMA), a novel cross-attention mechanism in which attention scores are defined as a function of the joint geometry of a query and multiple modality-specific keys. VMA computes the volume spanned by query and key vectors across multiple modalities, capturing joint multimodal dependencies beyond pairwise similarity, enabling native modeling of any-order modality interactions. We integrate VMA into our novel multimodal transformer architecture, named GRAMformer, explicitly designed to integrate any number of modalities. We evaluate the proposed model on multimodal learning tasks, demonstrating improved effectiveness and efficiency.