Data Consistency Approach to Model Validation
This addresses the lack of automatic validation tools for statistical models, which is crucial for scientific inference across various data types, though it appears incremental as it builds on existing validation concepts.
The paper tackles the problem of validating statistical modeling assumptions by introducing a general criterion to evaluate model consistency with observed data, achieving this by automatically assessing models' ability to generate similar data, as demonstrated on synthetic and real datasets.
In scientific inference problems, the underlying statistical modeling assumptions have a crucial impact on the end results. There exist, however, only a few automatic means for validating these fundamental modelling assumptions. The contribution in this paper is a general criterion to evaluate the consistency of a set of statistical models with respect to observed data. This is achieved by automatically gauging the models' ability to generate data that is similar to the observed data. Importantly, the criterion follows from the model class itself and is therefore directly applicable to a broad range of inference problems with varying data types, ranging from independent univariate data to high-dimensional time-series. The proposed data consistency criterion is illustrated, evaluated and compared to several well-established methods using three synthetic and two real data sets.