Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition
This work addresses a specific bottleneck in multimodal emotion recognition for applications like human-computer interaction, though it is incremental in nature.
The paper tackled the problem of frame-rate mismatch in audio-visual emotion recognition by proposing a Transformer-based framework with temporal alignment techniques, resulting in consistent improvements on CREMA-D and RAVDESS datasets.
Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.