Human-AI Interaction and User Satisfaction: Empirical Evidence from Online Reviews of AI Products
It provides large-scale empirical evidence on HAI principles for AI developers and researchers, though it is incremental as it applies existing guidelines to new data.
This study analyzed over 100,000 user reviews of AI products to investigate how Human-AI Interaction (HAI) dimensions affect user satisfaction, finding that sentiment on adaptability, customization, error recovery, and security positively correlates with satisfaction, and that engagement with HAI dimensions varies by job role but the satisfaction effect is consistent across roles.
Human-AI Interaction (HAI) guidelines and design principles have become increasingly important in both industry and academia to guide the development of AI systems that align with user needs and expectations. However, large-scale empirical evidence on how HAI principles shape user satisfaction in practice remains limited. This study addresses that gap by analyzing over 100,000 user reviews of AI-related products from G2, a leading review platform for business software and services. Based on widely adopted industry guidelines, we identify seven core HAI dimensions and examine their coverage and sentiment within the reviews. We find that the sentiment on four HAI dimensions-adaptability, customization, error recovery, and security-is positively associated with overall user satisfaction. Moreover, we show that engagement with HAI dimensions varies by professional background: Users with technical job roles are more likely to discuss system-focused aspects, such as reliability, while non-technical users emphasize interaction-focused features like customization and feedback. Interestingly, the relationship between HAI sentiment and overall satisfaction is not moderated by job role, suggesting that once an HAI dimension has been identified by users, its effect on satisfaction is consistent across job roles.