Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions
This research addresses the problem of user-AI interaction quality for designers of human-centered AI systems, though it is incremental as it builds on existing alignment concepts.
The study analyzed 25,000 Bing Copilot conversations to examine how AI agent responses align with user expertise, finding that 77% of responses were at proficient or expert levels, which correlated with positive user experience, while misalignment negatively impacted experience, especially for complex tasks.
Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise and the resulting impact on user experience along multiple dimensions. Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise (77% of conversations) which correlates with positive user experience regardless of the user's level of expertise. Misalignment, such that the agent responds at a level of expertise below that of the user, has a negative impact on overall user experience, with the impact more profound for more complex tasks. We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user. Our findings underscore the importance of alignment between user and AI when designing human-centered AI systems, to ensure satisfactory and productive interactions.