Classifier-guided Gradient Modulation for Enhanced Multimodal Learning
This addresses a common bottleneck in multimodal learning for researchers and practitioners by improving model performance and versatility, though it appears incremental as it builds on existing gradient modulation techniques.
The paper tackled the problem of multimodal learning models relying too heavily on one modality during training, which leads to inadequate use of other modalities, by proposing a novel method called Classifier-Guided Gradient Modulation (CGGM) that balances training by modulating both the magnitude and direction of gradients. The results show that CGGM outperforms all baselines and state-of-the-art methods consistently across four multimodal datasets covering classification, regression, and segmentation tasks.
Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based on which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients. To solve these problems, in this paper, we present a novel method to balance multimodal learning with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state-of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https://github.com/zrguo/CGGM.