Distributed Machine Learning through Heterogeneous Edge Systems
This addresses the challenge of efficient distributed training for AI applications on edge systems with varying computing capacities, though it is incremental as it builds on existing parameter server architectures.
The paper tackles the problem of distributed machine learning on heterogeneous edge devices by proposing ADSP, a parameter synchronization scheme that reduces waiting time and accelerates convergence, showing significant improvements in convergence time, scalability, and adaptability in experiments.
Many emerging AI applications request distributed machine learning (ML) among edge systems (e.g., IoT devices and PCs at the edge of the Internet), where data cannot be uploaded to a central venue for model training, due to their large volumes and/or security/privacy concerns. Edge devices are intrinsically heterogeneous in computing capacity, posing significant challenges to parameter synchronization for parallel training with the parameter server (PS) architecture. This paper proposes ADSP, a parameter synchronization scheme for distributed machine learning (ML) with heterogeneous edge systems. Eliminating the significant waiting time occurring with existing parameter synchronization models, the core idea of ADSP is to let faster edge devices continue training, while committing their model updates at strategically decided intervals. We design algorithms that decide time points for each worker to commit its model update, and ensure not only global model convergence but also faster convergence. Our testbed implementation and experiments show that ADSP outperforms existing parameter synchronization models significantly in terms of ML model convergence time, scalability and adaptability to large heterogeneity.