Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge
This addresses the challenge of AI-assisted decision-making in domains like social work and healthcare, where traditional evaluation methods fail, offering a novel training approach that is incremental in applying existing concepts to new contexts.
The paper tackles the problem of training humans to appropriately rely on AI predictions in complex real-world settings where ground truth labels are unavailable, by introducing a process-oriented notion called 'critical use' and conducting an experiment in child maltreatment screening. The result shows that novices, through accelerated practice, exhibited disagreement patterns similar to experienced workers by using qualitative case narratives unavailable to the AI.
A growing body of research has explored how to support humans in making better use of AI-based decision support, including via training and onboarding. Existing research has focused on decision-making tasks where it is possible to evaluate "appropriate reliance" by comparing each decision against a ground truth label that cleanly maps to both the AI's predictive target and the human decision-maker's goals. However, this assumption does not hold in many real-world settings where AI tools are deployed today (e.g., social work, criminal justice, and healthcare). In this paper, we introduce a process-oriented notion of appropriate reliance called critical use that centers the human's ability to situate AI predictions against knowledge that is uniquely available to them but unavailable to the AI model. To explore how training can support critical use, we conduct a randomized online experiment in a complex social decision-making setting: child maltreatment screening. We find that, by providing participants with accelerated, low-stakes opportunities to practice AI-assisted decision-making in this setting, novices came to exhibit patterns of disagreement with AI that resemble those of experienced workers. A qualitative examination of participants' explanations for their AI-assisted decisions revealed that they drew upon qualitative case narratives, to which the AI model did not have access, to learn when (not) to rely on AI predictions. Our findings open new questions for the study and design of training for real-world AI-assisted decision-making.