Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
This research addresses the critical gap in evaluating explanation effectiveness for ODQA systems in voice assistant contexts, aiming to improve user trust and decision-making for end-users of these systems.
This paper investigates the effectiveness of spoken versus visual explanations for open-domain question answering (ODQA) systems, finding that explanations derived from retrieved evidence passages can outperform calibrated confidence baselines across both modalities. Crucially, the optimal explanation strategy varies depending on whether it is delivered via a spoken or visual interface.
While research on explaining predictions of open-domain QA systems (ODQA) to users is gaining momentum, most works have failed to evaluate the extent to which explanations improve user trust. While few works evaluate explanations using user studies, they employ settings that may deviate from the end-user's usage in-the-wild: ODQA is most ubiquitous in voice-assistants, yet current research only evaluates explanations using a visual display, and may erroneously extrapolate conclusions about the most performant explanations to other modalities. To alleviate these issues, we conduct user studies that measure whether explanations help users correctly decide when to accept or reject an ODQA system's answer. Unlike prior work, we control for explanation modality, e.g., whether they are communicated to users through a spoken or visual interface, and contrast effectiveness across modalities. Our results show that explanations derived from retrieved evidence passages can outperform strong baselines (calibrated confidence) across modalities but the best explanation strategy in fact changes with the modality. We show common failure cases of current explanations, emphasize end-to-end evaluation of explanations, and caution against evaluating them in proxy modalities that are different from deployment.