Kernel Model Validation: How To Do It, And Why You Should Care
This work addresses the issue of unreliable uncertainty estimates in GP models for researchers and practitioners in uncertainty quantification, though it is incremental as it builds on existing validation methods.
The paper tackles the problem of probabilistic calibration in Gaussian Process (GP) models for uncertainty quantification, showing that calibration failures can degrade convergence in optimization algorithms like Targeted Adaptive Design, and proposes a formal kernel validation procedure to address this.
Gaussian Process (GP) models are popular tools in uncertainty quantification (UQ) because they purport to furnish functional uncertainty estimates that can be used to represent model uncertainty. It is often difficult to state with precision what probabilistic interpretation attaches to such an uncertainty, and in what way is it calibrated. Without such a calibration statement, the value of such uncertainty estimates is quite limited and qualitative. We motivate the importance of proper probabilistic calibration of GP predictions by describing how GP predictive calibration failures can cause degraded convergence properties in a target optimization algorithm called Targeted Adaptive Design (TAD). We discuss the interpretation of GP-generated uncertainty intervals in UQ, and how one may learn to trust them, through a formal procedure for covariance kernel validation that exploits the multivariate normal nature of GP predictions. We give simple examples of GP regression misspecified 1-dimensional models, and discuss the situation with respect to higher-dimensional models.