AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
This addresses the problem of slow training in heterogeneous collaborative learning environments, offering a practical solution for data holders, though it is incremental as it builds on existing pruning and collaborative learning techniques.
The paper tackles inefficiencies in multi-party collaborative learning due to heterogeneity by proposing AdaptCL, which dynamically prunes global models to match worker capabilities, achieving over 41% time savings on average and up to 6.2x speedup with minimal accuracy loss.
In multi-party collaborative learning, the parameter server sends a global model to each data holder for local training and then aggregates committed models globally to achieve privacy protection. However, both the dragger issue of synchronous collaborative learning and the staleness issue of asynchronous collaborative learning make collaborative learning inefficient in real-world heterogeneous environments. We propose a novel and efficient collaborative learning framework named AdaptCL, which generates an adaptive sub-model dynamically from the global base model for each data holder, without any prior information about worker capability. All workers (data holders) achieve approximately identical update time as the fastest worker by equipping them with capability-adapted pruned models. Thus the training process can be dramatically accelerated. Besides, we tailor the efficient pruned rate learning algorithm and pruning approach for AdaptCL. Meanwhile, AdaptCL provides a mechanism for handling the trade-off between accuracy and time overhead and can be combined with other techniques to accelerate training further. Empirical results show that AdaptCL introduces little computing and communication overhead. AdaptCL achieves time savings of more than 41\% on average and improves accuracy in a low heterogeneous environment. In a highly heterogeneous environment, AdaptCL achieves a training speedup of 6.2x with a slight loss of accuracy.