Same Performance, Hidden Bias: Evaluating Hypothesis- and Recommendation-Driven AI
This work highlights a hidden bias in AI decision support systems that could impact users in HCI and related fields, though it is incremental in focusing on process rather than performance.
The study examined how recommendation-driven versus hypothesis-driven AI interaction designs affect users' decision-making processes, finding that even with identical task performance, recommendation-driven designs lower evidence thresholds and introduce hidden biases in judgments, affecting both experts and novices equally.
The HCI community commonly evaluates decision support systems based on whether they improve task performance or promote appropriate user reliance. In this work, we look beyond decision outcomes to examine the process through which users develop decision-making strategies. Through a web-based experiment (N = 290) comparing recommendation-driven and hypothesis-driven interaction designs, and using Signal Detection Theory as a theoretical framework, we show that even when performance remains identical, recommendation-driven designs lower participants' thresholds for sufficient evidence and introduce a "hidden bias" in their judgments, resulting in a shifted distribution of errors. Furthermore, we find that experts are just as susceptible to these systemic shifts as novices. We conclude by advocating for a shift in focus: prioritizing decision processes and the preservation of stable evidence standards over performance and reliance alone.