Resource-Adaptive Federated Text Generation with Differential Privacy
This work is significant for organizations using federated learning with sensitive text data, as it enables the inclusion of resource-constrained clients in the generation of privacy-preserving synthetic datasets, mitigating data skew and the adverse effects of DP noise.
This paper addresses the challenge of generating differentially private (DP) synthetic text datasets in cross-silo federated learning (FL) where computational heterogeneity excludes weaker clients. The authors propose a two-phase framework that allows strong clients to perform DP federated finetuning while weak clients contribute via a lightweight DP voting mechanism, improving distribution alignment and downstream robustness under DP and heterogeneity.
In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client's data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.