Collaborative Learning via Prediction Consensus
This addresses the challenge of efficient and robust model improvement in collaborative settings, particularly for heterogeneous architectures, though it is incremental as it builds on existing distillation and consensus techniques.
The paper tackles the problem of collaborative learning where agents aim to improve their models by leveraging others' expertise, proposing a distillation-based method with a trust weighting scheme that achieves consensus on pseudo-labels for shared unlabeled data, resulting in significant performance boosts in the target domain.
We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging shared unlabeled auxiliary data, which is pseudo-labeled by the collective. Central to our method is a trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme is able to significantly boost the performance of individual models in the target domain from which the auxiliary data is sampled. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared to typical collaborative learning methods. At the same time, it can provably mitigate the negative impact of bad models on the collective.