Unexploited Information Value in Human-AI Collaboration
This work addresses the challenge of optimizing human-AI collaboration for decision-making tasks, though it appears incremental as it builds on existing models without introducing a new paradigm.
The paper tackles the problem of understanding how to improve human-AI team performance by analyzing unexploited information value, using a statistical decision theory model applied to a deepfake detection task to compare human, AI, and team performance.
Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistical decision theory to analyze human-AI collaboration from the perspective of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people's usage of the information and what information that the AI exploits well might be useful for improving human decisions.