Intrinsic uncertainties and where to find them
This addresses the problem of practical uncertainty quantification for ML practitioners by offering a framework that unifies and extends existing methods.
The paper tackles uncertainty estimation in machine learning by treating training hyperparameters as random variables and marginalizing them out to capture parameter space uncertainties, finding that certain marginalization combinations provide reliable uncertainty estimates without extensive tuning or large ensembles.
We introduce a framework for uncertainty estimation that both describes and extends many existing methods. We consider typical hyperparameters involved in classical training as random variables and marginalise them out to capture various sources of uncertainty in the parameter space. We investigate which forms and combinations of marginalisation are most useful from a practical point of view on standard benchmarking data sets. Moreover, we discuss how some marginalisations may produce reliable estimates of uncertainty without the need for extensive hyperparameter tuning and/or large-scale ensembling.