Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures
This work provides a crucial diagnostic tool for benchmark builders and researchers to accurately attribute errors in vision-language models to either perception or reasoning, preventing misattribution of extraction-stage fabrication to reasoning.
This paper introduces the render ceiling, a model-free method to separate perception from reasoning in vision-language models by rendering known objects and inverting the process. They applied this to 2,160 rendered crystal structures, finding that supplying exact geometry as text closed less than half the performance gap for 13 out of 14 models, and a supervised vision model without language achieved 0.8952 accuracy, outperforming all vision-language models.
Multimodal evaluations cannot say whether a vision-language model misread an image or misreasoned about it, because every existing method for separating the two places a second model in the loop. We introduce the render ceiling, a model-free reference for benchmarks built by rendering known objects: inverting the frozen cameras and re-solving cross-view correspondence recovers exactly the answer the images support. We prove the ceiling fails only through an enumerable set of projection coincidences and certify that set empty on 2,160 rendered crystal structures, so every point of a model's deficit belongs to the model. Across fourteen vision-language models, supplying exact geometry as text lifts every model yet closes under half the gap for thirteen, while a supervised vision model with no language component reads the same images at 0.8952, above every vision-language model. The instrument exposes extraction-stage fabrication that downstream accuracy would misattribute to reasoning, yields camera-placement rules for benchmark builders, and transfers to any benchmark with an invertible forward rendering.