MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization
For practitioners deploying multimodal models in varied real-world conditions, MER-DG offers a simple, architecture-agnostic fix to improve domain generalization.
Multimodal models often overfit to cross-modal co-occurrences in training environments, hurting generalization. MER-DG regularizes encoder features via entropy maximization, achieving ~5% improvement over standard fusion and ~2% over prior methods on EPIC-Kitchens and HAC benchmarks.
Deploying multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-Entropy Regularization for Domain Generalization (MER-DG), which maximizes the entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC benchmarks demonstrate average improvements of approximately 5% over standard fusion and approximately 2% over state-of-the-art methods.