StreamFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
This addresses a scalability problem for users deploying diffusion transformers in high-resolution image and video generation, representing an incremental improvement over existing sequence parallelism techniques.
The paper tackled the inefficiency of scaling diffusion transformer inference on GPUs due to suboptimal communication patterns and latency bottlenecks, and introduced StreamFusion, which improved performance by an average of 1.35x (up to 1.77x) over state-of-the-art methods.
Diffusion Transformers (DiTs) have gained increasing adoption in high-quality image and video generation. As demand for higher-resolution images and longer videos increases, single-GPU inference becomes inefficient due to increased latency and large activation sizes. Current frameworks employ sequence parallelism (SP) techniques such as Ulysses Attention and Ring Attention to scale inference. However, these implementations have three primary limitations: (1) suboptimal communication patterns for network topologies on modern GPU machines, (2) latency bottlenecks from all-to-all operations in inter-machine communication, and (3) GPU sender-receiver synchronization and computation overheads from using two-sided communication libraries. To address these issues, we present StreamFusion, a topology-aware efficient DiT serving engine. StreamFusion incorporates three key innovations: (1) a topology-aware sequence parallelism technique that accounts for inter- and intra-machine bandwidth differences, (2) Torus Attention, a novel SP technique enabling overlapping of inter-machine all-to-all operations with computation, and (3) a one-sided communication implementation that minimizes GPU sender-receiver synchronization and computation overheads. Our experiments demonstrate that StreamFusion outperforms the state-of-the-art approach by an average of $1.35\times$ (up to $1.77\times$).