Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources -- A Case Study on Federated Fine-tuning of LLaMA 2
This addresses the problem of enabling privacy-preserving collaborative training for organizations with sensitive data, though it appears incremental as it builds on existing federated learning and computing platforms.
The paper tackled the challenge of conducting secure federated learning across diverse computing environments by developing the APPFL framework, which demonstrated fine-tuning a LLaMA 2 7B model using cloud and supercomputing resources.
Federated learning enables multiple data owners to collaboratively train robust machine learning models without transferring large or sensitive local datasets by only sharing the parameters of the locally trained models. In this paper, we elaborate on the design of our Advanced Privacy-Preserving Federated Learning (APPFL) framework, which streamlines end-to-end secure and reliable federated learning experiments across cloud computing facilities and high-performance computing resources by leveraging Globus Compute, a distributed function as a service platform, and Amazon Web Services. We further demonstrate the use case of APPFL in fine-tuning a LLaMA 2 7B model using several cloud resources and supercomputers.