Feedback Effects in Repeat-Use Criminal Risk Assessments
This study highlights a critical flaw in the validation of criminal risk assessment tools, impacting practitioners and policymakers who rely on these tools for fair and accurate decision-making.
This paper investigates the implications of feedback effects in sequential scoring-decision processes within criminal risk assessments, demonstrating through simulation that even undetectable bias can amplify over sequential decisions, leading to observable group differences. The study concludes that current one-shot validation tests fail to capture these compounding effects.
In the criminal legal context, risk assessment algorithms are touted as data-driven, well-tested tools. Studies known as validation tests are typically cited by practitioners to show that a particular risk assessment algorithm has predictive accuracy, establishes legitimate differences between risk groups, and maintains some measure of group fairness in treatment. To establish these important goals, most tests use a one-shot, single-point measurement. Using a Polya Urn model, we explore the implication of feedback effects in sequential scoring-decision processes. We show through simulation that risk can propagate over sequential decisions in ways that are not captured by one-shot tests. For example, even a very small or undetectable level of bias in risk allocation can amplify over sequential risk-based decisions, leading to observable group differences after a number of decision iterations. Risk assessment tools operate in a highly complex and path-dependent process, fraught with historical inequity. We conclude from this study that these tools do not properly account for compounding effects, and require new approaches to development and auditing.