Beyond Forced Modality Balance: Intrinsic Information Budgets for Multimodal Learning
This addresses modality imbalance in multimodal models, which is an incremental improvement over existing balancing methods.
The paper tackles the problem of modality imbalance in multimodal learning, where a dominant modality overshadows weaker ones, by proposing IIBalance, a framework that uses Intrinsic Information Budgets to align modality contributions, resulting in consistent outperformance of state-of-the-art balancing methods on three benchmarks.
Multimodal models often converge to a dominant-modality solution, in which a stronger, faster-converging modality overshadows weaker ones. This modality imbalance causes suboptimal performance. Existing methods attempt to balance different modalities by reweighting gradients or losses. However, they overlook the fact that each modality has finite information capacity. In this work, we propose IIBalance, a multimodal learning framework that aligns the modality contributions with Intrinsic Information Budgets (IIB). We propose a task-grounded estimator of each modality's IIB, transforming its capacity into a global prior over modality contributions. Anchored by the highest-budget modality, we design a prototype-based relative alignment mechanism that corrects semantic drift only when weaker modalities deviate from their budgeted potential, rather than forcing imitation. During inference, we propose a probabilistic gating module that integrates the global budgets with sample-level uncertainty to generate calibrated fusion weights. Experiments on three representative benchmarks demonstrate that IIBalance consistently outperforms state-of-the-art balancing methods and achieves better utilization of complementary modality cues. Our code is available at: https://github.com/XiongZechang/IIBalance.