FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models
This framework addresses the problem of enabling small and medium-sized enterprises to adopt LLMs by reducing resource barriers and data privacy concerns, though it appears incremental as it builds on existing federated learning and parameter-efficient fine-tuning methods.
The paper tackles the challenges of high computing resource consumption and scattered data for training large language models (LLMs) by proposing FATE-LLM, an industrial-grade federated learning framework that enables federated learning for LLMs, promotes efficient training, and protects intellectual property and data privacy.
Large Language Models (LLMs), such as ChatGPT, LLaMA, GLM, and PaLM, have exhibited remarkable performances across various tasks in recent years. However, LLMs face two main challenges in real-world applications. One challenge is that training LLMs consumes vast computing resources, preventing LLMs from being adopted by small and medium-sized enterprises with limited computing resources. Another is that training LLM requires a large amount of high-quality data, which are often scattered among enterprises. To address these challenges, we propose FATE-LLM, an industrial-grade federated learning framework for large language models. FATE-LLM (1) facilitates federated learning for large language models (coined FedLLM); (2) promotes efficient training of FedLLM using parameter-efficient fine-tuning methods; (3) protects the intellectual property of LLMs; (4) preserves data privacy during training and inference through privacy-preserving mechanisms. We release the code of FATE-LLM at https://github.com/FederatedAI/FATE-LLM to facilitate the research of FedLLM and enable a broad range of industrial applications.