The Non-IID Data Quagmire of Decentralized Machine Learning
This addresses a pervasive challenge in decentralized ML for applications with distributed data, but it is incremental as it builds on existing understanding with experimental insights.
The paper tackles the problem of decentralized machine learning on non-IID data, showing that skewed data labels cause significant accuracy loss across various applications and models, and presents SkewScout, a system that adapts communication frequency to mitigate this loss, with group normalization recovering accuracy.
Many large-scale machine learning (ML) applications need to perform decentralized learning over datasets generated at different devices and locations. Such datasets pose a significant challenge to decentralized learning because their different contexts result in significant data distribution skew across devices/locations. In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. Our study shows that: (i) skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms; (ii) the problem is particularly challenging for DNN models with batch normalization; and (iii) the degree of data skew is a key determinant of the difficulty of the problem. Based on these findings, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the accuracy loss of batch normalization.