Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approach
This work addresses the trade-off between personalization and federation in federated learning for foundation models, which is an incremental improvement for scenarios with non-IID data and limited user data.
The paper tackles the challenge of fine-tuning federated foundation models for small user groups or specialized scenarios with limited data by proposing a bi-level personalization framework that combines client-level fine-tuning with server-level personalized aggregation based on task vectors, demonstrating effectiveness through experiments on benchmark datasets.
Federated foundation models represent a new paradigm to jointly fine-tune pre-trained foundation models across clients. It is still a challenge to fine-tune foundation models for a small group of new users or specialized scenarios, which typically involve limited data compared to the large-scale data used in pre-training. In this context, the trade-off between personalization and federation becomes more sensitive. To tackle these, we proposed a bi-level personalization framework for federated fine-tuning on foundation models. Specifically, we conduct personalized fine-tuning on the client-level using its private data, and then conduct a personalized aggregation on the server-level using similar users measured by client-specific task vectors. Given the personalization information gained from client-level fine-tuning, the server-level personalized aggregation can gain group-wise personalization information while mitigating the disturbance of irrelevant or interest-conflict clients with non-IID data. The effectiveness of the proposed algorithm has been demonstrated by extensive experimental analysis in benchmark datasets.