Think-Aloud Verbalizations for Identifying User Experience Problems: Effects of Language Proficiency with Chinese Non-Native English Speakers
This addresses the generalizability of UX analysis methods to non-native English speakers, an incremental step for UX practitioners and AI tool designers.
The study investigated whether think-aloud verbalization patterns used to identify user experience problems, previously established with native English speakers, also apply to Chinese non-native English speakers, finding that both groups had similar verbalization categories, encountered similar problems, and showed comparable patterns indicating UX issues.
Subtle patterns in users' think-aloud (TA) verbalizations (i.e., utterances) are shown to be telltale signs of user experience (UX) problems and used to build artificial intelligence (AI) models or AI-assisted tools to help UX evaluators identify UX problems automatically or semi-automatically. Despite the potential of such verbalization patterns, they were uncovered with native English speakers. As most people who speak English are non-native speakers, it is important to investigate whether similar patterns exist in non-native English speakers' TA verbalizations. As a first step to answer this question, we conducted think-aloud usability testing with Chinese non-native English speakers and native English speakers using three common TA protocols. We compared their verbalizations and UX problems that they encountered to understand the effects of language and TA protocols. Our findings show that both language groups had similar amounts and proportions of verbalization categories, encountered similar problems, and had similar verbalization patterns that indicate UX problems. Furthermore, TA protocols did not significantly affect the correlations between verbalizations and problems. Based on the findings, we present three design implications for UX practitioners and the design of AI-assisted analysis tools.