Federated Learning with Layer Skipping: Efficient Training of Large Language Models for Healthcare NLP
This addresses privacy-preserving collaborative training for healthcare NLP, though it appears incremental as it builds on existing federated learning and fine-tuning methods.
The paper tackles the problem of high communication costs and data heterogeneity in federated learning for large language models in healthcare NLP by proposing Layer-Skipping Federated Learning, which reduces communication costs by approximately 70% while maintaining performance within 2% of centralized training.
Federated learning (FL) enables collaborative model training across organizations without sharing raw data, addressing crucial privacy concerns in healthcare natural language processing (NLP). However, training large language models (LLMs) in federated settings faces significant challenges, including communication overhead and data heterogeneity. We propose Layer-Skipping Federated Learning, where only selected layers of a pre-trained LLM are fine-tuned across clients while others remain frozen. Applied to LLaMA 3.2-1B, our approach reduces communication costs by approximately 70% while maintaining performance within 2% of centralized training. We evaluate our method on clinical NER and classification tasks using i2b2 and MIMIC-III datasets. Our experiments demonstrate that Layer-Skipping FL outperforms competitive baselines, handles non-IID clinical data distributions effectively, and shows robustness when combined with differential privacy. This approach represents a practical solution for privacy-preserving collaborative learning in healthcare NLP.