Model-Free Assessment of Simulator Fidelity via Quantile Curves
This addresses the critical need for model-free assessment of simulator fidelity, enabling statistical inference and risk comparisons across simulators, though it is incremental in applying existing statistical concepts to a new AI context.
The paper tackles the problem of quantifying the sim-to-real gap in generative AI models by constructing confidence sets for latent distribution parameters and estimating quantile functions to provide a risk profile, demonstrating its utility by evaluating the alignment of four major LLMs with human populations on the WorldValueBench dataset.
As generative AI models are increasingly used to simulate real-world systems, quantifying the ``sim-to-real'' gap is critical. The distributional discrepancy between real and simulated outputs is a random variable driven by the stochastic input scenario. A fundamental challenge is that for any given input, the ground-truth and simulated output distributions are only observable through finite batches of samples, often of heterogeneous sizes. This renders standard predictive inference methods inapplicable, as they seek to quantify uncertainty in observable outputs rather than their underlying population parameters. To address this, we construct confidence sets for these latent parameters and use them to derive a robust proxy for the sim-to-real discrepancy. We then estimate the quantile function of this proxy to provide a comprehensive risk profile of the simulator. Our method is model-agnostic and handles general output spaces, such as categorical survey responses and continuous multi-dimensional sensor data. By rigorously accounting for sampling error, the resulting risk profile supports statistical inference for the real output distribution in a new scenario, the calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators. We demonstrate the practical utility of this method by evaluating the alignment of four major LLMs with human populations on the WorldValueBench dataset.