Selective Attention Federated Learning: Improving Privacy and Efficiency for Clinical Text Classification
This work addresses privacy and efficiency issues in federated learning for clinical text classification, offering a domain-specific solution that is incremental in nature.
The paper tackled the challenges of communication overhead and model privacy in federated learning for large language models in healthcare by introducing Selective Attention Federated Learning (SAFL), which dynamically fine-tunes attention-critical transformer layers, achieving competitive performance with centralized models while improving communication efficiency and privacy preservation.
Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation.