Posterior Dispersion Indices
This work addresses the need for better evaluation metrics in probabilistic modeling for researchers and practitioners, though it appears incremental as it builds on existing uncertainty analysis methods.
The authors tackled the problem of evaluating probabilistic models beyond predictive accuracy by proposing posterior dispersion indices (PDI) to analyze datapoints in relation to posterior uncertainty, and they demonstrated its effectiveness in identifying model mismatch patterns in three real-world datasets.
Probabilistic modeling is cyclical: we specify a model, infer its posterior, and evaluate its performance. Evaluation drives the cycle, as we revise our model based on how it performs. This requires a metric. Traditionally, predictive accuracy prevails. Yet, predictive accuracy does not tell the whole story. We propose to evaluate a model through posterior dispersion. The idea is to analyze how each datapoint fares in relation to posterior uncertainty around the hidden structure. We propose a family of posterior dispersion indices (PDI) that capture this idea. A PDI identifies rich patterns of model mismatch in three real data examples: voting preferences, supermarket shopping, and population genetics.