A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings
It provides a comprehensive overview for researchers and developers working on scalable ML systems, but it is incremental as a survey rather than presenting new methods.
This survey tackles the challenge of high computational demands in deep learning by exploring distributed learning approaches across cloud, mobile, and edge settings, analyzing partitioning schemes and trade-offs to optimize resource use.
In the era of deep learning (DL), convolutional neural networks (CNNs), and large language models (LLMs), machine learning (ML) models are becoming increasingly complex, demanding significant computational resources for both inference and training stages. To address this challenge, distributed learning has emerged as a crucial approach, employing parallelization across various devices and environments. This survey explores the landscape of distributed learning, encompassing cloud and edge settings. We delve into the core concepts of data and model parallelism, examining how models are partitioned across different dimensions and layers to optimize resource utilization and performance. We analyze various partitioning schemes for different layer types, including fully connected, convolutional, and recurrent layers, highlighting the trade-offs between computational efficiency, communication overhead, and memory constraints. This survey provides valuable insights for future research and development in this rapidly evolving field by comparing and contrasting distributed learning approaches across diverse contexts.