Evaluating Bayesian Model Visualisations
This work addresses the need for better tools to support decision-makers in understanding and using Bayesian models, though it is incremental in improving evaluation practices.
The paper tackles the problem of evaluating Bayesian model visualizations by proposing a quantitative evaluation protocol and a software framework to standardize and reproduce assessments, illustrated through a user study on interactive Boxplots and Hypothetical Outcome Plots.
Probabilistic models inform an increasingly broad range of business and policy decisions ultimately made by people. Recent algorithmic, computational, and software framework development progress facilitate the proliferation of Bayesian probabilistic models, which characterise unobserved parameters by their joint distribution instead of point estimates. While they can empower decision makers to explore complex queries and to perform what-if-style conditioning in theory, suitable visualisations and interactive tools are needed to maximise users' comprehension and rational decision making under uncertainty. In this paper, propose a protocol for quantitative evaluation of Bayesian model visualisations and introduce a software framework implementing this protocol to support standardisation in evaluation practice and facilitate reproducibility. We illustrate the evaluation and analysis workflow on a user study that explores whether making Boxplots and Hypothetical Outcome Plots interactive can increase comprehension or rationality and conclude with design guidelines for researchers looking to conduct similar studies in the future.