Examining user reviews of conversational systems: a case study of Alexa skills
This work addresses user experience problems for developers and researchers of conversational systems, but it is incremental as it applies existing review analysis methods to a new domain.
The study analyzed over 2,800 Alexa skills to identify user issues, finding that most skills receive fewer than 50 reviews and that content, integration, errors, and regression are top complaints, with differences noted compared to mobile apps.
Conversational systems use spoken language to interact with their users. Although conversational systems, such as Amazon Alexa, are becoming common and afford interesting functionalities, there is little known about the issues users of these systems face. In this paper, we study user reviews of more than 2,800 Alexa skills to understand the characteristics of the reviews and issues that are raised in them. Our results suggest that most skills receive less than 50 reviews. Our qualitative study of user reviews using open coding resulted in identifying 16 types of issues in the user reviews. Issues related to the content, integration with online services and devices, error, and regression are top issues raised by the users. Our results also indicate differences in volume and types of complaints by users when compared with more traditional mobile applications. We discuss the implication of our results for practitioners and researchers.