A Diachronic Perspective on User Trust in AI under Uncertainty
This addresses the problem of building trustworthy AI for users in human-AI collaboration by highlighting the impact of calibration on trust, though it is incremental in exploring specific trust dynamics.
The study investigated how user trust in AI systems evolves after trust-eroding events, such as incorrect predictions with inaccurate confidence estimates, using a betting game. It found that even a few such instances significantly damage trust and collaboration performance, with slow recovery, and that different types of miscalibration have varying negative effects.
In a human-AI collaboration, users build a mental model of the AI system based on its reliability and how it presents its decision, e.g. its presentation of system confidence and an explanation of the output. Modern NLP systems are often uncalibrated, resulting in confidently incorrect predictions that undermine user trust. In order to build trustworthy AI, we must understand how user trust is developed and how it can be regained after potential trust-eroding events. We study the evolution of user trust in response to these trust-eroding events using a betting game. We find that even a few incorrect instances with inaccurate confidence estimates damage user trust and performance, with very slow recovery. We also show that this degradation in trust reduces the success of human-AI collaboration and that different types of miscalibration -- unconfidently correct and confidently incorrect -- have different negative effects on user trust. Our findings highlight the importance of calibration in user-facing AI applications and shed light on what aspects help users decide whether to trust the AI system.