Measuring Voice UX Quantitatively: A Rapid Review
This work addresses the urgent problem of quantifying voice UX for researchers and developers, but it is incremental as it reviews existing literature and proposes frameworks without new empirical results.
The paper tackled the lack of consensus in measuring user experience (UX) for voice-based systems by surveying ACM and IEEE literature, finding little agreement and overreliance on lab work and unvalidated scales. It resulted in two high-level descriptive frameworks to guide future research and standardize measurements.
Computer voice is experiencing a renaissance through the growing popularity of voice-based interfaces, agents, and environments. Yet, how to measure the user experience (UX) of voice-based systems remains an open and urgent question, especially given that their form factors and interaction styles tend to be non-visual, intangible, and often considered disembodied or "body-less." As a first step, we surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX. Our findings show that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales. In response, we offer two high-level descriptive frameworks for guiding future research, developing standardized instruments, and informing ongoing review work. Our work highlights the current strengths and weaknesses of voice UX research and charts a path towards measuring voice UX in a more comprehensive way.