Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices
This work addresses efficiency issues in federated learning for fine-tuning language models on heterogeneous devices, offering an incremental improvement over existing parameter-efficient methods.
The paper tackles the challenges of resource constraints and system heterogeneity in federated fine-tuning by proposing LEGEND, a novel LoRA-based framework that optimizes LoRA depth and rank distribution for heterogeneous devices, achieving a speedup of 1.5-2.8x and saving communication costs by about 42.3% while maintaining target accuracy.
Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8$\times$ and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.