ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
This addresses communication bottlenecks for resource-constrained clients in federated learning of large language models, representing an incremental improvement over existing methods.
The paper tackles the high communication latency in federated block coordinate descent for large language models by proposing ParaBlock, which uses parallel communication and computation threads, achieving the same convergence rate as standard methods and significantly improving communication efficiency in empirical evaluations on fine-tuning tasks.
Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.