Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offs
For practitioners training multimodal large language models, this work demonstrates that data scheduling is a first-order design variable that can significantly influence the balance between general understanding, reasoning, and OCR capabilities.
The paper investigates how the temporal organization of heterogeneous supervision data affects capability trade-offs in multimodal instruction tuning. It finds that curriculum training (building general understanding and reasoning before OCR-intensive tasks) yields the best overall trade-off and strongest structured reasoning performance, while balanced sampling favors OCR but weakens broader capabilities.
Recent multimodal large language models (MLLMs) perform strongly on general visual understanding, diagram and chart reasoning, and document-centric perception. However, these abilities are learned from heterogeneous supervision sources with very different task structures and learning demands, and the effect of their temporal organization during training remains underexplored. We study whether data organization affects the trade-off among general understanding, structured reasoning, and fine-grained OCR/document understanding in multimodal instruction tuning. To isolate this factor, we use a controlled three-stage training framework in which the backbone, trainable modules, and optimization pipeline are fixed across all runs, and only the temporal arrangement of post-alignment supervision is changed. We compare four strategies: direct mixture, curriculum training, balanced sampling, and reverse curriculum. Experiments on general visual instruction following, diagram reasoning, chart reasoning, scene-text question answering, and document question answering show that data organization is a first-order design variable in multimodal adaptation. Curriculum training gives the best overall trade-off and the strongest structured reasoning performance. Balanced sampling is better for OCR-oriented capability but weakens the broader capability balance. Reverse curriculum performs worst in both final performance and optimization stability. Training-dynamics analysis further suggests that building general understanding and reasoning before introducing OCR-intensive supervision leads to smoother optimization and faster convergence. These findings highlight data scheduling as an explicit design dimension for multimodal model adaptation.