Efficient Zero-Order Federated Finetuning of Language Models for Resource-Constrained Devices
This work addresses the problem of efficient and private model adaptation for edge computing, representing an incremental improvement in optimization methods.
The paper tackles the challenge of federated fine-tuning of large language models on resource-constrained edge devices by proposing FedSPZO, a method that reduces computation overhead by 2.5 to 7 times compared to existing zero-order techniques.
Federated fine-tuning offers a promising approach for tuning Large Language Models (LLMs) on edge devices while preserving data privacy. However, fine-tuning these models on edge devices remains challenging due to high memory, communication, and computational demands. Zero-order optimization with task alignment provides a potential solution, enabling fine-tuning with inference-level memory requirements but requires a longer convergence time. In this paper, we propose Federated Split-Perturbation Zero-order Optimization (FedSPZO) that divides the network into two blocks, applying a different number of perturbations per block in a computationally effective way, achieving faster convergence. Our evaluation shows a $2.5 - 7\times $ reduction in computation overhead compared to zero-order state of the art techniques in federated learning.