RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
This addresses the problem of incomplete data in multimodal emotion recognition for applications like human-computer interaction, though it appears incremental as it builds on existing IMER methods with hybrid techniques.
The paper tackles the Incomplete Multimodal Emotion Recognition (IMER) challenge by proposing RoHyDR, a framework that recovers missing or corrupted data at multiple levels using diffusion-based generation and adversarial learning, achieving robust performance and outperforming state-of-the-art methods on two benchmarks.
Multimodal emotion recognition analyzes emotions by combining data from multiple sources. However, real-world noise or sensor failures often cause missing or corrupted data, creating the Incomplete Multimodal Emotion Recognition (IMER) challenge. In this paper, we propose Robust Hybrid Diffusion Recovery (RoHyDR), a novel framework that performs missing-modality recovery at unimodal, multimodal, feature, and semantic levels. For unimodal representation recovery of missing modalities, RoHyDR exploits a diffusion-based generator to generate distribution-consistent and semantically aligned representations from Gaussian noise, using available modalities as conditioning. For multimodal fusion recovery, we introduce adversarial learning to produce a realistic fused multimodal representation and recover missing semantic content. We further propose a multi-stage optimization strategy that enhances training stability and efficiency. In contrast to previous work, the hybrid diffusion and adversarial learning-based recovery mechanism in RoHyDR allows recovery of missing information in both unimodal representation and multimodal fusion, at both feature and semantic levels, effectively mitigating performance degradation caused by suboptimal optimization. Comprehensive experiments conducted on two widely used multimodal emotion recognition benchmarks demonstrate that our proposed method outperforms state-of-the-art IMER methods, achieving robust recognition performance under various missing-modality scenarios. Our code will be made publicly available upon acceptance.