Establishing Appropriate Trust via Critical States
This addresses the problem of trust and supervision in human-robot interaction for end-users, though it is incremental as it builds on existing mental model concepts.
The paper tackles the challenge of humans understanding learned neural network policies in robotics by identifying critical states where specific actions are crucial, and shows that revealing these states helps users decide when to deploy policies and take control, with user studies indicating improved informed decision-making.
In order to effectively interact with or supervise a robot, humans need to have an accurate mental model of its capabilities and how it acts. Learned neural network policies make that particularly challenging. We propose an approach for helping end-users build a mental model of such policies. Our key observation is that for most tasks, the essence of the policy is captured in a few critical states: states in which it is very important to take a certain action. Our user studies show that if the robot shows a human what its understanding of the task's critical states is, then the human can make a more informed decision about whether to deploy the policy, and if she does deploy it, when she needs to take control from it at execution time.