Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis
This addresses the problem of data scarcity for AI developers by providing a scalable data synthesis method, though it appears incremental as it builds on existing MLLM and VLM techniques.
The paper tackles the challenge of costly and time-consuming real-world data collection for AI-generated content by proposing Follow-Your-Instruction, an MLLM-driven framework that automatically synthesizes high-quality 2D, 3D, and 4D data, with results showing that the synthetic data significantly boosts the performance of existing baseline models.
With the growing demands of AI-generated content (AIGC), the need for high-quality, diverse, and scalable data has become increasingly crucial. However, collecting large-scale real-world data remains costly and time-consuming, hindering the development of downstream applications. While some works attempt to collect task-specific data via a rendering process, most approaches still rely on manual scene construction, limiting their scalability and accuracy. To address these challenges, we propose Follow-Your-Instruction, a Multimodal Large Language Model (MLLM)-driven framework for automatically synthesizing high-quality 2D, 3D, and 4D data. Our \textbf{Follow-Your-Instruction} first collects assets and their associated descriptions through multimodal inputs using the MLLM-Collector. Then it constructs 3D layouts, and leverages Vision-Language Models (VLMs) for semantic refinement through multi-view scenes with the MLLM-Generator and MLLM-Optimizer, respectively. Finally, it uses MLLM-Planner to generate temporally coherent future frames. We evaluate the quality of the generated data through comprehensive experiments on the 2D, 3D, and 4D generative tasks. The results show that our synthetic data significantly boosts the performance of existing baseline models, demonstrating Follow-Your-Instruction's potential as a scalable and effective data engine for generative intelligence.