A flexible framework for communication-efficient machine learning: from HPC to IoT
This addresses communication bottlenecks for machine learning practitioners working with varied systems from HPC to IoT, though it is incremental as it builds on existing compression methods.
The paper tackles the problem of communication inefficiency in machine learning across diverse system architectures by proposing a flexible framework that adapts compression levels per iteration to maximize objective improvement per communicated bit, with experiments showing significant increases in communication efficiency on state-of-the-art compression schemes.
With the increasing scale of machine learning tasks, it has become essential to reduce the communication between computing nodes. Early work on gradient compression focused on the bottleneck between CPUs and GPUs, but communication-efficiency is now needed in a variety of different system architectures, from high-performance clusters to energy-constrained IoT devices. In the current practice, compression levels are typically chosen before training and settings that work well for one task may be vastly suboptimal for another dataset on another architecture. In this paper, we propose a flexible framework which adapts the compression level to the true gradient at each iteration, maximizing the improvement in the objective function that is achieved per communicated bit. Our framework is easy to adapt from one technology to the next by modeling how the communication cost depends on the compression level for the specific technology. Theoretical results and practical experiments indicate that the automatic tuning strategies significantly increase communication efficiency on several state-of-the-art compression schemes.