Peer-to-Peer Learning + Consensus with Non-IID Data
This addresses performance instability for distributed edge devices in collaborative training, but appears incremental as it builds on existing P2PL methods.
The paper tackled the problem of model parameter drift and performance oscillations in peer-to-peer deep learning with non-IID data, and demonstrated that their novel approach, P2PL with Affinity, dampens these oscillations without extra communication cost.
Peer-to-peer deep learning algorithms are enabling distributed edge devices to collaboratively train deep neural networks without exchanging raw training data or relying on a central server. Peer-to-Peer Learning (P2PL) and other algorithms based on Distributed Local-Update Stochastic/mini-batch Gradient Descent (local DSGD) rely on interleaving epochs of training with distributed consensus steps. This process leads to model parameter drift/divergence amongst participating devices in both IID and non-IID settings. We observe that model drift results in significant oscillations in test performance evaluated after local training and consensus phases. We then identify factors that amplify performance oscillations and demonstrate that our novel approach, P2PL with Affinity, dampens test performance oscillations in non-IID settings without incurring any additional communication cost.