(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court
This work addresses the problem of ensuring accountable AI use in legal systems for public defenders and defendants, but it is incremental as it builds on existing research on contestability.
The study investigated the challenges public defenders face in contesting computational forensic software (CFS) like facial recognition in court, finding difficulties in navigating its development, overcoming non-critical perceptions, and gathering expertise, with recommendations provided to improve contestability.
Accountable use of AI systems in high-stakes settings relies on making systems contestable. In this paper we study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court. We present findings from interviews with 17 people in the U.S. public defense community to understand their perceptions of and experiences scrutinizing computational forensic software (CFS) -- automated decision systems that the government uses to convict and incarcerate, such as facial recognition, gunshot detection, and probabilistic genotyping tools. We find that our participants faced challenges assessing and contesting CFS reliability due to difficulties (a) navigating how CFS is developed and used, (b) overcoming judges and jurors' non-critical perceptions of CFS, and (c) gathering CFS expertise. To conclude, we provide recommendations that center the technical, social, and institutional context to better position interventions such as performance evaluations to support contestability in practice.