Federated Learning via Input-Output Collaborative Distillation
This work addresses privacy concerns for users in federated learning by eliminating the need for parameter sharing or auxiliary data, though it is incremental as it builds on existing distillation methods.
The authors tackled the problem of privacy leakage and data dependency in federated learning by proposing a data-free framework using input-output collaborative distillation, which achieved notable privacy-utility trade-offs in experiments on image classification and segmentation tasks under heterogeneous settings.
Federated learning (FL) is a machine learning paradigm in which distributed local nodes collaboratively train a central model without sharing individually held private data. Existing FL methods either iteratively share local model parameters or deploy co-distillation. However, the former is highly susceptible to private data leakage, and the latter design relies on the prerequisites of task-relevant real data. Instead, we propose a data-free FL framework based on local-to-central collaborative distillation with direct input and output space exploitation. Our design eliminates any requirement of recursive local parameter exchange or auxiliary task-relevant data to transfer knowledge, thereby giving direct privacy control to local users. In particular, to cope with the inherent data heterogeneity across locals, our technique learns to distill input on which each local model produces consensual yet unique results to represent each expertise. Our proposed FL framework achieves notable privacy-utility trade-offs with extensive experiments on image classification and segmentation tasks under various real-world heterogeneous federated learning settings on both natural and medical images.