Structured Cooperative Learning with Graphical Model Priors
This addresses the challenge of efficient and accurate personalized learning in decentralized settings, such as edge computing, with incremental improvements over prior decentralized learning approaches.
The paper tackles the problem of training personalized models on decentralized devices with limited local data by proposing Structured Cooperative Learning (SCooL), which uses graphical model priors to generate cooperation graphs for coordinating mutual learning, resulting in the highest accuracy and significantly improved communication efficiency compared to existing methods.
We study how to train personalized models for different tasks on decentralized devices with limited local data. We propose "Structured Cooperative Learning (SCooL)", in which a cooperation graph across devices is generated by a graphical model prior to automatically coordinate mutual learning between devices. By choosing graphical models enforcing different structures, we can derive a rich class of existing and novel decentralized learning algorithms via variational inference. In particular, we show three instantiations of SCooL that adopt Dirac distribution, stochastic block model (SBM), and attention as the prior generating cooperation graphs. These EM-type algorithms alternate between updating the cooperation graph and cooperative learning of local models. They can automatically capture the cross-task correlations among devices by only monitoring their model updating in order to optimize the cooperation graph. We evaluate SCooL and compare it with existing decentralized learning methods on an extensive set of benchmarks, on which SCooL always achieves the highest accuracy of personalized models and significantly outperforms other baselines on communication efficiency. Our code is available at https://github.com/ShuangtongLi/SCooL.