Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients
This addresses the need for collaborative model personalization in AI applications like Agentic AI, moving beyond simplified assumptions to handle real-world task diversity, though it is incremental in advancing personalized federated learning.
The paper tackled the problem of personalizing models in federated learning under realistic scenarios with data and model heterogeneity, proposing FedMosaic which outperformed state-of-the-art methods in personalization and generalization across 40 multi-modal tasks.
As AI becomes more personal, e.g., Agentic AI, there is an increasing need for personalizing models for various use cases. Personalized federated learning (PFL) enables each client to collaboratively leverage other clients' knowledge for better adaptation to the task of interest, without privacy risks. Despite its potential, existing PFL methods remain confined to rather simplified scenarios where data and models are the same across clients. To move towards realistic scenarios, we propose FedMosaic, a method that jointly addresses data and model heterogeneity with a task-relevance-aware model aggregation strategy to reduce parameter interference, and a dimension-invariant module that enables knowledge sharing across heterogeneous architectures without huge computational cost. To mimic the real-world task diversity, we propose a multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time. The empirical study shows that FedMosaic outperforms the state-of-the-art PFL methods, excelling in both personalization and generalization capabilities under challenging, realistic scenarios.