Evaluating User Experiences in Mixed Reality
This work aims to improve research methods for MR user studies, but it is incremental as it focuses on initiating discussions rather than presenting new solutions.
The paper addresses the lack of standardized measurement tools for evaluating user experiences in mixed reality (MR), highlighting challenges in adapting methods from other domains and proposing a workshop to discuss and standardize assessment methods.
Measure user experience in MR (i.e., AR/VR) user studies is essential. Researchers apply a wide range of measuring methods using objective (e.g., biosignals, time logging), behavioral (e.g., gaze direction, movement amplitude), and subjective (e.g., standardized questionnaires) metrics. Many of these measurement instruments were adapted from use-cases outside of MR but have not been validated for usage in MR experiments. However, researchers are faced with various challenges and design alternatives when measuring immersive experiences. These challenges become even more diverse when running out-of-the lab studies. Measurement methods of VR experience recently received much attention. For example, research has started embedding questionnaires in the VE for various applications, allowing users to stay closer to the ongoing experience while filling out the survey. However, there is a diversity in the interaction methods and practices on how the assessment procedure is conducted. This diversity in methods underlines a missing shared agreement of standardized measurement tools for VR experiences. AR research strongly orients on the research methods from VR, e.g., using the same type of subjective questionnaires. However, some crucial technical differences require careful considerations during the evaluation. This workshop at CHI 2021 provides a foundation to exchange expertise and address challenges and opportunities of research methods in MR user studies. By this, our workshop launches a discussion of research methods that should lead to standardizing assessment methods in MR user studies. The outcomes of the workshop will be aggregated into a collective special issue journal article.