3FM: Multi-modal Meta-learning for Federated Tasks
This addresses challenges in federated learning for clients with varying modality availability, but it appears incremental as it builds on existing meta-learning and FL methods.
The paper tackles the problem of modality heterogeneity and missing data in federated learning by proposing a meta-learning framework for multimodal tasks, achieving better performance than a baseline on an augmented MNIST dataset with audio and sign language data.
We present a novel approach in the domain of federated learning (FL), particularly focusing on addressing the challenges posed by modality heterogeneity, variability in modality availability across clients, and the prevalent issue of missing data. We introduce a meta-learning framework specifically designed for multimodal federated tasks. Our approach is motivated by the need to enable federated models to robustly adapt when exposed to new modalities, a common scenario in FL where clients often differ in the number of available modalities. The effectiveness of our proposed framework is demonstrated through extensive experimentation on an augmented MNIST dataset, enriched with audio and sign language data. We demonstrate that the proposed algorithm achieves better performance than the baseline on a subset of missing modality scenarios with careful tuning of the meta-learning rates. This is a shortened report, and our work will be extended and updated soon.