Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation
This addresses the problem of domain-specific reasoning limitations in VLMs for AI researchers and practitioners, representing a strong but incremental advance through improved data integration.
The authors tackled the limited generalization of visual reasoning in VLMs by curating a multi-domain dataset and training Vision-G1 with a data selection and curriculum strategy, achieving state-of-the-art performance across benchmarks and outperforming models like GPT-4o and Gemini-1.5 Flash.
Despite their success, current training pipelines for reasoning VLMs focus on a limited range of tasks, such as mathematical and logical reasoning. As a result, these models face difficulties in generalizing their reasoning capabilities to a wide range of domains, primarily due to the scarcity of readily available and verifiable reward data beyond these narrowly defined areas. Moreover, integrating data from multiple domains is challenging, as the compatibility between domain-specific datasets remains uncertain. To address these limitations, we build a comprehensive RL-ready visual reasoning dataset from 46 data sources across 8 dimensions, covering a wide range of tasks such as infographic, mathematical, spatial, cross-image, graphic user interface, medical, common sense and general science. We propose an influence function based data selection and difficulty based filtering strategy to identify high-quality training samples from this dataset. Subsequently, we train the VLM, referred to as Vision-G1, using multi-round RL with a data curriculum to iteratively improve its visual reasoning capabilities. Our model achieves state-of-the-art performance across various visual reasoning benchmarks, outperforming similar-sized VLMs and even proprietary models like GPT-4o and Gemini-1.5 Flash. The model, code and dataset are publicly available at https://github.com/yuh-zha/Vision-G1.