Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection
This addresses the problem of modal bias for researchers and practitioners in multi-modal federated learning, offering an incremental improvement over existing client selection methods.
The paper tackles the modality imbalance problem in multi-modal federated learning, which causes global bias and reduces effectiveness, by proposing a Balanced Modality Selection framework (BMSFed) that enhances local training and selects diverse modalities, achieving superior performance on audio-visual, colored-gray, and front-back datasets.
Selecting proper clients to participate in each federated learning (FL) round is critical to effectively harness a broad range of distributed data. Existing client selection methods simply consider the mining of distributed uni-modal data, yet, their effectiveness may diminish in multi-modal FL (MFL) as the modality imbalance problem not only impedes the collaborative local training but also leads to a severe global modality-level bias. We empirically reveal that local training with a certain single modality may contribute more to the global model than training with all local modalities. To effectively exploit the distributed multiple modalities, we propose a novel Balanced Modality Selection framework for MFL (BMSFed) to overcome the modal bias. On the one hand, we introduce a modal enhancement loss during local training to alleviate local imbalance based on the aggregated global prototypes. On the other hand, we propose the modality selection aiming to select subsets of local modalities with great diversity and achieving global modal balance simultaneously. Our extensive experiments on audio-visual, colored-gray, and front-back datasets showcase the superiority of BMSFed over baselines and its effectiveness in multi-modal data exploitation.