Quantifying Fidelity: A Decisive Feature Approach to Comparing Synthetic and Real Imagery
This addresses the challenge of ensuring synthetic data reliability for autonomous vehicle safety testing, though it is incremental by extending existing fidelity measures with a SUT-specific approach.
The paper tackled the problem that pixel-level fidelity in synthetic data does not ensure reliable transfer to real-world autonomous vehicle testing, proposing a new metric called Decisive Feature Fidelity (DFF) to measure agreement in decision evidence across domains, with experiments on 2126 matched pairs showing DFF reveals overlooked discrepancies and improves fidelity without sacrificing output value.
Virtual testing using synthetic data has become a cornerstone of autonomous vehicle (AV) safety assurance. Despite progress in improving visual realism through advanced simulators and generative AI, recent studies reveal that pixel-level fidelity alone does not ensure reliable transfer from simulation to the real world. What truly matters is whether the system-under-test (SUT) bases its decisions on consistent decision evidence in both real and simulated environments, not just whether images "look real" to humans. To this end this paper proposes a behavior-grounded fidelity measure by introducing Decisive Feature Fidelity (DFF), a new SUT-specific metric that extends the existing fidelity spectrum to capture mechanism parity, that is, agreement in the model-specific decisive evidence that drives the SUT's decisions across domains. DFF leverages explainable-AI methods to identify and compare the decisive features driving the SUT's outputs for matched real-synthetic pairs. We further propose estimators based on counterfactual explanations, along with a DFF-guided calibration scheme to enhance simulator fidelity. Experiments on 2126 matched KITTI-VirtualKITTI2 pairs demonstrate that DFF reveals discrepancies overlooked by conventional output-value fidelity. Furthermore, results show that DFF-guided calibration improves decisive-feature and input-level fidelity without sacrificing output value fidelity across diverse SUTs.