A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
For researchers and engineers designing distributed training systems for LLMs, this work provides a more accurate evaluation framework that reveals the importance of communication in pipeline schedule performance.
The paper introduces a tabular schedule abstraction and a unified methodology to compare pipeline-parallel LLM training schedules, showing that communication can negate structural advantages from bubble analysis alone. Under their assumptions, GPipe and 1F1B are runtime-equivalent, with 1F1B having lower activation-memory peak; Chimera is advantageous only at low microbatch counts and in communication-favorable regimes.
Pipeline parallelism is a key technique for distributed training of large language models because it reduces per-device parameter and activation memory. However, comparing pipeline schedules is difficult: analytical models expose structural quantities such as bubble ratios, while end-to-end hardware experiments are costly and system-specific. In this work, we introduce a tabular schedule abstraction and a unified multi-abstraction methodology that connects formula-based reasoning, idealized schedule tables, and communication-aware execution simulation. Using this framework, we compare GPipe, 1F1B, Chimera, and Hanayo in its restricted regime across multiple modeled system configurations. Our results show that schedule rankings are not abstraction-invariant: communication can negate structural advantages suggested by bubble analysis alone. Under the assumptions considered here, GPipe and 1F1B are runtime-equivalent, but 1F1B achieves a lower activation-memory peak. Chimera is advantageous mainly at low microbatch counts and in communication-favorable regimes, while Hanayo is effective in its intended restricted operating point but remains sensitive to network bottlenecks. We further study an asymmetric Chimera-style placement, which does not reduce the global peak memory requirement but reveals limited runtime gains in shallow pipelines. Overall, pipeline schedule quality is meaningful only in the context of the modeled execution environment.