MrTF: Model Refinery for Transductive Federated Learning
This addresses the challenge of data heterogeneity and inference for new data in federated learning, particularly for real-world pilot projects requiring privacy protection, though it appears incremental by extending existing FL methods.
The paper tackles the problem of making inferences for newly-collected data in federated learning under privacy constraints by proposing a transductive federated learning paradigm that incorporates test samples into training, resulting in improved predictions and robust model fusion as verified by experiments.
We consider a real-world scenario in which a newly-established pilot project needs to make inferences for newly-collected data with the help of other parties under privacy protection policies. Current federated learning (FL) paradigms are devoted to solving the data heterogeneity problem without considering the to-be-inferred data. We propose a novel learning paradigm named transductive federated learning (TFL) to simultaneously consider the structural information of the to-be-inferred data. On the one hand, the server could use the pre-available test samples to refine the aggregated models for robust model fusion, which tackles the data heterogeneity problem in FL. On the other hand, the refinery process incorporates test samples into training and could generate better predictions in a transductive manner. We propose several techniques including stabilized teachers, rectified distillation, and clustered label refinery to facilitate the model refinery process. Abundant experimental studies verify the superiorities of the proposed \underline{M}odel \underline{r}efinery framework for \underline{T}ransductive \underline{F}ederated learning (MrTF). The source code is available at \url{https://github.com/lxcnju/MrTF}.