Aggregating Low Rank Adapters in Federated Fine-tuning
This work addresses efficiency challenges in federated learning for fine-tuning large models, but appears incremental as it builds on existing LoRA methods.
The paper tackles the problem of high computational and communication costs in federated fine-tuning of large language models by proposing a novel aggregation method for low-rank adapters (LoRA). It compares this method with existing approaches and evaluates performance on GLUE benchmark datasets, though no concrete numbers are provided in the abstract.
Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a novel aggregation method and compare it with different existing aggregation methods of low rank adapters trained in a federated fine-tuning of large machine learning models and evaluate their performance with respect to selected GLUE benchmark datasets.