Mixup Helps Understanding Multimodal Video Better
This addresses modality imbalance in multimodal video understanding for tasks like action recognition, though it is incremental as it builds on existing Mixup techniques.
The paper tackles the problem of multimodal video models overfitting to strong modalities by proposing Multimodal Mixup (MM) and Balanced Multimodal Mixup (B-MM), which apply Mixup at the feature level and dynamically adjust mixing ratios, respectively, resulting in improved generalization and robustness across several datasets.
Multimodal video understanding plays a crucial role in tasks such as action recognition and emotion classification by combining information from different modalities. However, multimodal models are prone to overfitting strong modalities, which can dominate learning and suppress the contributions of weaker ones. To address this challenge, we first propose Multimodal Mixup (MM), which applies the Mixup strategy at the aggregated multimodal feature level to mitigate overfitting by generating virtual feature-label pairs. While MM effectively improves generalization, it treats all modalities uniformly and does not account for modality imbalance during training. Building on MM, we further introduce Balanced Multimodal Mixup (B-MM), which dynamically adjusts the mixing ratios for each modality based on their relative contributions to the learning objective. Extensive experiments on several datasets demonstrate the effectiveness of our methods in improving generalization and multimodal robustness.