BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
This addresses the challenge of effectively integrating multiple modalities in machine learning models, particularly when noise is present, which is important for applications like sentiment analysis and clinical classification.
The paper tackles the problem of multimodal model integration where additional modalities may introduce noise rather than complementary information, proposing BTW, a non-parametric weighting framework that dynamically adjusts modality importance during training. The method significantly improves regression performance and multiclass classification accuracy in experiments on sentiment regression and clinical classification.
Mixture-of-Experts (MoE) models have become increasingly powerful in multimodal learning by enabling modular specialization across modalities. However, their effectiveness remains unclear when additional modalities introduce more noise than complementary information. Existing approaches, such as the Partial Information Decomposition, struggle to scale beyond two modalities and lack the resolution needed for instance-level control. We propose Beyond Two-modality Weighting (BTW), a bi-level, non-parametric weighting framework that combines instance-level Kullback-Leibler (KL) divergence and modality-level mutual information (MI) to dynamically adjust modality importance during training. Our method does not require additional parameters and can be applied to an arbitrary number of modalities. Specifically, BTW computes per-example KL weights by measuring the divergence between each unimodal and the current multimodal prediction, and modality-wide MI weights by estimating global alignment between unimodal and multimodal outputs. Extensive experiments on sentiment regression and clinical classification demonstrate that our method significantly improves regression performance and multiclass classification accuracy.