Boba: Authoring and Visualizing Multiverse Analyses
This work addresses the problem of managing and interpreting complex multiverse analyses for data analysts, though it is incremental as it builds on existing multiverse analysis concepts with new software tools.
The authors tackled the challenge of specifying and interpreting multiverse analyses, which involve evaluating all reasonable analytic decisions in parallel, by developing Boba, an integrated domain-specific language and visual analysis system. The result is a tool that allows analysts to write shared code once, generate scripts for all analysis paths, and visually assess robustness and decision impacts through case studies.
Multiverse analysis is an approach to data analysis in which all "reasonable" analytic decisions are evaluated in parallel and interpreted collectively, in order to foster robustness and transparency. However, specifying a multiverse is demanding because analysts must manage myriad variants from a cross-product of analytic decisions, and the results require nuanced interpretation. We contribute Boba: an integrated domain-specific language (DSL) and visual analysis system for authoring and reviewing multiverse analyses. With the Boba DSL, analysts write the shared portion of analysis code only once, alongside local variations defining alternative decisions, from which the compiler generates a multiplex of scripts representing all possible analysis paths. The Boba Visualizer provides linked views of model results and the multiverse decision space to enable rapid, systematic assessment of consequential decisions and robustness, including sampling uncertainty and model fit. We demonstrate Boba's utility through two data analysis case studies, and reflect on challenges and design opportunities for multiverse analysis software.