SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning
This work addresses the critical challenges of client heterogeneity and communication overhead in federated split learning for LLMs, which is important for enabling privacy-preserving fine-tuning across diverse devices.
SplitFT introduces an adaptive federated split learning system for LLM fine-tuning that allows heterogeneous clients to set different cut layers based on their resources and performance, and reduces communication overhead by lowering LoRA rank at the cut layer. Experiments show it outperforms state-of-the-art methods in fine-tuning time efficiency and model performance on popular benchmarks.
Federated Split Learning has been identified as an efficient approach to address the computational resource constraints of clients in classical federated learning, while guaranteeing data privacy for distributed model training across data owners. However, it faces some critical challenges when such a training strategy meets large language models (LLMs) for fine-tuning. Such challenges include setting the cutlayer adaptively across different clients to address the data and device heterogeneity issues, which affect the system performance significantly. In addition, efficiently reducing the communication overhead during the fine-tuning procedure is also another challenge. No work tries to address these challenges. To bridge this gap, we propose SplitTF, an adaptive federated split learning system for LLMs fine-tuning. SplitFT enables different clients to set different cut layers according to their computation resources and trained model performance. SplitFT also proposes to reduce the LoRA rank in cutlayer to reduce the communication overhead. In addition to simulating the heterogeneous data in real-world applications for our proposed split federated learning system, we propose a length-based Dirichlet approach to divide the training data into different clients. Extensive experimental results show that our proposed approach outperforms the state-of-the-art approach for fine-tuning time efficiency and model performance based on various popular benchmarks.