Wizard of Errors: Introducing and Evaluating Machine Learning Errors in Wizard of Oz Studies
This work addresses a specific problem for designers of ML-enabled solutions by providing a tool to improve user experience testing, but it is incremental as it builds on existing Wizard of Oz methods.
The paper tackles the problem of simulating machine learning errors in Wizard of Oz studies for user experience testing by introducing Wizard of Errors (WoE), a tool that allows designers to incorporate ML errors, and finds that it helps identify challenges in achieving realistic error representation.
When designing Machine Learning (ML) enabled solutions, designers often need to simulate ML behavior through the Wizard of Oz (WoZ) approach to test the user experience before the ML model is available. Although reproducing ML errors is essential for having a good representation, they are rarely considered. We introduce Wizard of Errors (WoE), a tool for conducting WoZ studies on ML-enabled solutions that allows simulating ML errors during user experience assessment. We explored how this system can be used to simulate the behavior of a computer vision model. We tested WoE with design students to determine the importance of considering ML errors in design, the relevance of using descriptive error types instead of confusion matrix, and the suitability of manual error control in WoZ studies. Our work identifies several challenges, which prevent realistic error representation by designers in such studies. We discuss the implications of these findings for design.