NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics
For researchers developing physically-grounded VLMs, this work provides a standardized diagnostic paradigm to evaluate and improve faithfulness in spatial and physical reasoning.
The paper diagnoses and calibrates vision-language models (VLMs) in quantitative reasoning for kinematic physics, finding that models fail to identify visual preconditions or utilize necessary physical laws, and proposes a novel calibration method and metrics to improve confidence reliability.
The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.