Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality
This addresses a practical challenge in multi-modal AI systems for applications like AVQA, though it is incremental as it builds on existing AVQA methods to handle missing data.
The paper tackles the problem of performance degradation in Audio-Visual Question Answering (AVQA) when audio or visual modalities are missing due to real-world issues like device malfunctions, proposing a framework that ensures robust performance by generating missing modal information and enhancing audio-visual features, achieving accurate answers even with incomplete inputs.
Recent Audio-Visual Question Answering (AVQA) methods rely on complete visual and audio input to answer questions accurately. However, in real-world scenarios, issues such as device malfunctions and data transmission errors frequently result in missing audio or visual modality. In such cases, existing AVQA methods suffer significant performance degradation. In this paper, we propose a framework that ensures robust AVQA performance even when a modality is missing. First, we propose a Relation-aware Missing Modal (RMM) generator with Relation-aware Missing Modal Recalling (RMMR) loss to enhance the ability of the generator to recall missing modal information by understanding the relationships and context among the available modalities. Second, we design an Audio-Visual Relation-aware (AVR) diffusion model with Audio-Visual Enhancing (AVE) loss to further enhance audio-visual features by leveraging the relationships and shared cues between the audio-visual modalities. As a result, our method can provide accurate answers by effectively utilizing available information even when input modalities are missing. We believe our method holds potential applications not only in AVQA research but also in various multi-modal scenarios.