WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
This addresses OoD detection for machine learning models, but it appears incremental as it builds on existing likelihood-based methods.
The paper tackles the problem of Out-of-Distribution (OoD) errors in high-dimensional data by proposing Generative Ensembles to robustify density-based OoD detection through epistemic uncertainty estimation, noting that WAIC performs well in practice despite theoretical limitations.
Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate the training data. One proposal to scale OoD detection to high-dimensional data is to learn a tractable likelihood approximation of the training distribution, and use it to reject unlikely inputs. However, likelihood models on natural data are themselves susceptible to OoD errors, and even assign large likelihoods to samples from other datasets. To mitigate this problem, we propose Generative Ensembles, which robustify density-based OoD detection by way of estimating epistemic uncertainty of the likelihood model. We present a puzzling observation in need of an explanation -- although likelihood measures cannot account for the typical set of a distribution, and therefore should not be suitable on their own for OoD detection, WAIC performs surprisingly well in practice.