My Actions Speak Louder Than Your Words: When User Behavior Predicts Their Beliefs about Agents' Attributes
This addresses the problem of biased user perceptions in AI agent evaluations, which is incremental but important for improving human-AI interaction models.
The study found that users who experienced better outcomes in a human-agent interaction rated the agent as more capable, benevolent, and honest than those with worse outcomes, even when outcomes were due to user behavior, with ratings differing by up to 30% on average.
An implicit expectation of asking users to rate agents, such as an AI decision-aid, is that they will use only relevant information -- ask them about an agent's benevolence, and they should consider whether or not it was kind. Behavioral science, however, suggests that people sometimes use irrelevant information. We identify an instance of this phenomenon, where users who experience better outcomes in a human-agent interaction systematically rated the agent as having better abilities, being more benevolent, and exhibiting greater integrity in a post hoc assessment than users who experienced worse outcome -- which were the result of their own behavior -- with the same agent. Our analyses suggest the need for augmentation of models so that they account for such biased perceptions as well as mechanisms so that agents can detect and even actively work to correct this and similar biases of users.