ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics
This addresses the problem of inefficient communication schedules for researchers and practitioners in distributed deep learning, offering a scalable solution for heterogeneous networks, though it appears incremental as it builds on existing scheduling methods.
The paper tackles the performance bottleneck of collective communications in large DNN models by presenting ForestColl, a tool that generates throughput-optimal schedules for any network topology, achieving significant improvements over vendor-optimized libraries and other state-of-the-art techniques in evaluations on AMD and NVIDIA clusters.
As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient communication schedules is challenging, given today's heterogeneous and diverse network fabrics. We present ForestColl, a tool that generates throughput-optimal schedules for any network topology. ForestColl constructs broadcast/aggregation spanning trees as the communication schedule, achieving theoretical optimality. Its schedule generation runs in polynomial time and is highly scalable. ForestColl supports any network fabric, including both switching fabrics and direct accelerator connections. We evaluated ForestColl on AMD MI250 and NVIDIA DGX A100 & H100 clusters. ForestColl showed significant improvements over the vendors' own optimized communication libraries across various settings and in LLM training. ForestColl also outperformed other state-of-the-art schedule generation techniques with both more efficient generated schedules and substantially faster generation speed.