SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data
This addresses the problem of poor generalization and communication inefficiency in decentralized training for distributed AI systems, representing an incremental improvement over existing methods.
The paper tackled the challenges of decentralized deep learning with heterogeneous data and high communication costs by proposing SADDLe, a sharpness-aware algorithm, which improved test accuracy by 1-20% and maintained robustness with only a 1% drop under 4x compression.
Decentralized training enables learning with distributed datasets generated at different locations without relying on a central server. In realistic scenarios, the data distribution across these sparsely connected learning agents can be significantly heterogeneous, leading to local model over-fitting and poor global model generalization. Another challenge is the high communication cost of training models in such a peer-to-peer fashion without any central coordination. In this paper, we jointly tackle these two-fold practical challenges by proposing SADDLe, a set of sharpness-aware decentralized deep learning algorithms. SADDLe leverages Sharpness-Aware Minimization (SAM) to seek a flatter loss landscape during training, resulting in better model generalization as well as enhanced robustness to communication compression. We present two versions of our approach and conduct extensive experiments to show that SADDLe leads to 1-20% improvement in test accuracy compared to other existing techniques. Additionally, our proposed approach is robust to communication compression, with an average drop of only 1% in the presence of up to 4x compression.