Robust Training of Federated Models with Extremely Label Deficiency
This work addresses a specific bottleneck in federated learning for distributed data with limited labels, offering an incremental improvement over existing methods.
The paper tackles the problem of gradient conflicts in federated semi-supervised learning due to label deficiency by proposing Twin-sight, a twin-model paradigm that enhances mutual guidance between supervised and unsupervised models, resulting in significant performance improvements over state-of-the-art methods on four benchmark datasets.
Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. Advanced FSSL methods predominantly focus on training a single model on each client. However, this approach could lead to a discrepancy between the objective functions of labeled and unlabeled data, resulting in gradient conflicts. To alleviate gradient conflict, we propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. In particular, Twin-sight concurrently trains a supervised model with a supervised objective function while training an unsupervised model using an unsupervised objective function. To enhance the synergy between these two models, Twin-sight introduces a neighbourhood-preserving constraint, which encourages the preservation of the neighbourhood relationship among data features extracted by both models. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings, demonstrating the efficacy of the proposed Twin-sight.