Flexible Parallel Learning in Edge Scenarios: Communication, Computational and Energy Cost
This work addresses the need for efficient parallel learning in edge computing environments, offering a practical solution for IoT and fog scenarios, though it appears incremental as it builds on existing distributed learning approaches.
The paper tackles the problem of combining data and model parallelism for distributed machine learning in fog- and IoT-based edge scenarios, presenting a flexible parallel learning (FPL) framework that achieves an excellent trade-off among computational, communication, and energy costs while maintaining learning performance.
Traditionally, distributed machine learning takes the guise of (i) different nodes training the same model (as in federated learning), or (ii) one model being split among multiple nodes (as in distributed stochastic gradient descent). In this work, we highlight how fog- and IoT-based scenarios often require combining both approaches, and we present a framework for flexible parallel learning (FPL), achieving both data and model parallelism. Further, we investigate how different ways of distributing and parallelizing learning tasks across the participating nodes result in different computation, communication, and energy costs. Our experiments, carried out using state-of-the-art deep-network architectures and large-scale datasets, confirm that FPL allows for an excellent trade-off among computational (hence energy) cost, communication overhead, and learning performance.