Federated Customization of Large Models: Approaches, Experiments, and Insights
This addresses the problem of customizing large models in federated learning settings for privacy-preserving applications, but it is incremental as it adapts existing methods.
The paper tackles federated customization of large models by reviewing techniques and conducting experiments on federated prefix-tuning, showing it achieves performance close to centralized approaches with competitive results in efficiency and robustness.
In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.