SPHINX: A Synthetic Environment for Visual Perception and Reasoning
This work addresses the problem of assessing and improving multimodal reasoning in AI systems, particularly for researchers in computer vision and AI, by providing a new benchmark, though it is incremental as it builds on existing synthetic environment and reinforcement learning approaches.
The authors tackled the challenge of evaluating visual perception and reasoning in AI by creating Sphinx, a synthetic environment that generates puzzles with verifiable solutions, and found that state-of-the-art models like GPT-5 achieve only 51.1% accuracy, below human performance, but reinforcement learning with verifiable rewards improved accuracy and transferred to other benchmarks.
We present Sphinx, a synthetic environment for visual perception and reasoning that targets core cognitive primitives. Sphinx procedurally generates puzzles using motifs, tiles, charts, icons, and geometric primitives, each paired with verifiable ground-truth solutions, enabling both precise evaluation and large-scale dataset construction. The benchmark covers 25 task types spanning symmetry detection, geometric transformations, spatial reasoning, chart interpretation, and sequence prediction. Evaluating recent large vision-language models (LVLMs) shows that even state-of-the-art GPT-5 attains only 51.1% accuracy, well below human performance. Finally, we demonstrate that reinforcement learning with verifiable rewards (RLVR) substantially improves model accuracy on these tasks and yields gains on external visual reasoning benchmarks, highlighting its promise for advancing multimodal reasoning.