Exploring Selective Layer Fine-Tuning in Federated Learning
This work addresses computational efficiency and adaptability in federated learning for clients with varying resources, though it is incremental as it builds on existing fine-tuning and FL methods.
The paper tackles the problem of efficiently fine-tuning foundation models in federated learning under limited resources by exploring selective layer fine-tuning, and it proposes a strategic layer selection method that improves model convergence and adapts to client heterogeneity, with experiments on image and text datasets demonstrating its effectiveness over baselines.
Federated learning (FL) has emerged as a promising paradigm for fine-tuning foundation models using distributed data in a privacy-preserving manner. Under limited computational resources, clients often find it more practical to fine-tune a selected subset of layers, rather than the entire model, based on their task-specific data. In this study, we provide a thorough theoretical exploration of selective layer fine-tuning in FL, emphasizing a flexible approach that allows the clients to adjust their selected layers according to their local data and resources. We theoretically demonstrate that the layer selection strategy has a significant impact on model convergence in two critical aspects: the importance of selected layers and the heterogeneous choices across clients. Drawing from these insights, we further propose a strategic layer selection method that utilizes local gradients and regulates layer selections across clients. The extensive experiments on both image and text datasets demonstrate the effectiveness of the proposed strategy compared with several baselines, highlighting its advances in identifying critical layers that adapt to the client heterogeneity and training dynamics in FL.