Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making
This addresses the problem of misplaced trust in ML decision-aid systems for users, highlighting risks in human-AI collaboration, but it is incremental as it builds on existing trust research.
The study measured how people's trust in ML recommendations varies by expertise and system information, finding that people often trust incorrect ML advice even when they perform well or are given low-confidence signals, and that system information generally increases trust.
ML decision-aid systems are increasingly common on the web, but their successful integration relies on people trusting them appropriately: they should use the system to fill in gaps in their ability, but recognize signals that the system might be incorrect. We measured how people's trust in ML recommendations differs by expertise and with more system information through a task-based study of 175 adults. We used two tasks that are difficult for humans: comparing large crowd sizes and identifying similar-looking animals. Our results provide three key insights: (1) People trust incorrect ML recommendations for tasks that they perform correctly the majority of the time, even if they have high prior knowledge about ML or are given information indicating the system is not confident in its prediction; (2) Four different types of system information all increased people's trust in recommendations; and (3) Math and logic skills may be as important as ML for decision-makers working with ML recommendations.