Towards Trustworthy Amortized Bayesian Model Comparison
This work addresses the reliability of model comparison for researchers in Bayesian statistics, but it is incremental as it builds on existing amortized methods with specific enhancements.
The paper tackled the problem of unreliable neural surrogates in amortized Bayesian model comparison under model misspecification by supplementing simulation-based training with a self-consistency loss on real data. The result showed that this approach improved calibration when using analytic likelihoods but offered limited gains with neural surrogate likelihoods.
Amortized Bayesian model comparison (BMC) enables fast probabilistic ranking of models via simulation-based training of neural surrogates. However, the reliability of neural surrogates deteriorates when simulation models are misspecified - the very case where model comparison is most needed. Thus, we supplement simulation-based training with a self-consistency (SC) loss on unlabeled real data to improve BMC estimates under empirical distribution shifts. Using a numerical experiment and two case studies with real data, we compare amortized evidence estimates with and without SC against analytic or bridge sampling benchmarks. SC improves calibration under model misspecification when having access to analytic likelihoods. However, it offers limited gains with neural surrogate likelihoods, making it most practical for trustworthy BMC when likelihoods are exact.