Improving Multimodal Learning Balance and Sufficiency through Data Remixing
This addresses modality imbalance and insufficiency in multimodal models, which is an incremental improvement over existing methods that fail to achieve both sufficiency and balance.
The paper tackles the problem of modality laziness and clash in multimodal learning by proposing a data remixing method that decouples and reassembles data to improve balance and sufficiency, resulting in accuracy improvements of approximately 6.50% on CREMAD and 3.41% on Kinetic-Sounds without extra computational cost.
Different modalities hold considerable gaps in optimization trajectories, including speeds and paths, which lead to modality laziness and modality clash when jointly training multimodal models, resulting in insufficient and imbalanced multimodal learning. Existing methods focus on enforcing the weak modality by adding modality-specific optimization objectives, aligning their optimization speeds, or decomposing multimodal learning to enhance unimodal learning. These methods fail to achieve both unimodal sufficiency and multimodal balance. In this paper, we, for the first time, address both concerns by proposing multimodal Data Remixing, including decoupling multimodal data and filtering hard samples for each modality to mitigate modality imbalance; and then batch-level reassembling to align the gradient directions and avoid cross-modal interference, thus enhancing unimodal learning sufficiency. Experimental results demonstrate that our method can be seamlessly integrated with existing approaches, improving accuracy by approximately 6.50%$\uparrow$ on CREMAD and 3.41%$\uparrow$ on Kinetic-Sounds, without training set expansion or additional computational overhead during inference. The source code is available at https://github.com/MatthewMaxy/Remix_ICML2025.