Assessing model error in counterfactual worlds
This work addresses the challenge of assessing model reliability for decision-makers in planning and scenario analysis, though it is incremental as it builds on existing methods for error estimation.
The paper tackles the problem of retrospectively evaluating counterfactual scenario projections by focusing on model miscalibration rather than scenario deviation, presenting three approaches to estimate model error in counterfactual worlds and demonstrating their benefits and limitations in a simulation experiment.
Counterfactual scenario modeling exercises that ask "what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.