FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models
For practitioners training large-scale vision-language models, FlowTrain addresses the inefficiency of distributed training by decoupling heterogeneous modules, offering a practical solution to improve throughput and resource utilization.
FlowTrain introduces a flow-based decoupled training framework for vision-language models that separates encoder and backbone execution via a unified memory pool, achieving over 50% MFU and up to 1.7x throughput improvement, narrowing the efficiency gap to LLM-only training.
Industrial-grade distributed training of vision-language models (VLMs) remains far less efficient than that of unimodal LLMs. Existing solutions either follow a monolithic design that assigns uniform parallelism to heterogeneous modules or adopt a disaggregated deployment that separates modules while executing them as a batch-synchronized pipeline. In this paper, we highlight that the above solutions are still not sufficient, and VLM training can be further decoupled. To this end, we present FlowTrain, a flow-based decoupled training framework that reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool. The encoder and backbone can progress independently over a global virtual address space. Since this execution decoupling fundamentally changes the optimization objective of allocation and scheduling, FlowTrain further introduces a heterogeneous parallel allocator that assigns module-specific parallelism strategies by solving a throughput matching problem. The dynamic packing scheduler is used to construct balanced microbatches at runtime according to the actual LLM-side computation cost. Extensive experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7x throughput improvement, narrowing the efficiency gap to LLM-only training.