2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support
This addresses the need for deeper insights into AI decision support pitfalls for fields adopting these technologies, though it is incremental in modeling these effects.
The paper tackles the problem of understanding the effects of AI-assisted decision making by introducing the 2-Step Agent framework, which uses Bayesian causal inference to model how predictions influence beliefs and downstream outcomes, showing through simulations that a single misaligned prior belief can lead to worse outcomes than no support.
Across a growing number of fields, human decision making is supported by predictions from AI models. However, we still lack a deep understanding of the effects of adoption of these technologies. In this paper, we introduce a general computational framework, the 2-Step Agent, which models the effects of AI-assisted decision making. Our framework uses Bayesian methods for causal inference to model 1) how a prediction on a new observation affects the beliefs of a rational Bayesian agent, and 2) how this change in beliefs affects the downstream decision and subsequent outcome. Using this framework, we show by simulations how a single misaligned prior belief can be sufficient for decision support to result in worse downstream outcomes compared to no decision support. Our results reveal several potential pitfalls of AI-driven decision support and highlight the need for thorough model documentation and proper user training.