DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism
This addresses the challenge of scaling long-context MLLM training efficiently for AI researchers and practitioners, representing an incremental improvement over existing parallelism methods.
The paper tackles the problem of inefficient training of Multimodal Large Language Models (MLLMs) due to data heterogeneity and static parallelism, proposing Dynamic Hybrid Parallelism (DHP) to adaptively reconfigure communication groups and parallelism degrees, resulting in up to 1.36× speedup in training throughput while maintaining near-linear scaling efficiency.
Scaling long-context capabilities is crucial for Multimodal Large Language Models (MLLMs). However, real-world multimodal datasets are extremely heterogeneous. Existing training frameworks predominantly rely on static parallelism strategies, which suffer from severe load imbalance, redundant communication, and suboptimal hardware utilization under data heterogeneity. In this work, we propose Dynamic Hybrid Parallelism (DHP), an efficient parallelism strategy that adaptively reconfigures communication groups and parallelism degrees during MLLM training. We generalize the non-power-of-two parallelism degrees and develop a polynomial-time algorithm to generate near-optimal parallelism strategies with only millisecond-level overhead per training batch. DHP is able to maintain high hardware efficiency even under extreme data variability. Experimental results demonstrate that DHP significantly outperforms Megatron-LM and DeepSpeed, achieving up to 1.36 $\times$ speedup in training throughput while maintaining near-linear scaling efficiency across large-scale NPU clusters.