Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence
This addresses the issue of point-wise reliability in predictive distributions for real-world decision-making, representing an incremental improvement over existing calibration metrics like ECE.
The paper tackles the problem of neural networks having overconfident and misaligned predictive distributions by proposing a stronger condition called conditional congruence and a new metric, Conditional Congruence Error (CCE), to assess probabilistic fit, showing that CCE exhibits correctness, monotonicity, reliability, and robustness in high-dimensional regression tasks.
While significant progress has been made in specifying neural networks capable of representing uncertainty, deep networks still often suffer from overconfidence and misaligned predictive distributions. Existing approaches for measuring this misalignment are primarily developed under the framework of calibration, with common metrics such as Expected Calibration Error (ECE). However, calibration can only provide a strictly marginal assessment of probabilistic alignment. Consequently, calibration metrics such as ECE are $\textit{distribution-wise}$ measures and cannot diagnose the $\textit{point-wise}$ reliability of individual inputs, which is important for real-world decision-making. We propose a stronger condition, which we term $\textit{conditional congruence}$, for assessing probabilistic fit. We also introduce a metric, Conditional Congruence Error (CCE), that uses conditional kernel mean embeddings to estimate the distance, at any point, between the learned predictive distribution and the empirical, conditional distribution in a dataset. We perform several high dimensional regression tasks and show that CCE exhibits four critical properties: $\textit{correctness}$, $\textit{monotonicity}$, $\textit{reliability}$, and $\textit{robustness}$.