MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models
This addresses efficient and personalized model adaptation in federated learning for LLMs, but it is incremental as it builds on existing parameter-efficient techniques.
The paper tackles federated fine-tuning of large language models by proposing a multi-task learning method that uses Low-Rank Adaptation to reduce computational overhead, achieving lower local loss for clients while maintaining global performance.
In this paper, we introduce a method for fine-tuning Large Language Models (LLMs), inspired by Multi-Task learning in a federated manner. Our approach leverages the structure of each client's model and enables a learning scheme that considers other clients' tasks and data distribution. To mitigate the extensive computational and communication overhead often associated with LLMs, we utilize a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), reducing the number of trainable parameters. Experimental results, with different datasets and models, demonstrate the proposed method's effectiveness compared to existing frameworks for federated fine-tuning of LLMs in terms of average and local performances. The proposed scheme outperforms existing baselines by achieving lower local loss for each client while maintaining comparable global performance.