Evdoxia Taka

Muffy Calder, Marion Oswald, Elizabeth McClory-Tiarks et al.

There is growing interest in the use of Large Language Models (LLMs) in policing, but there are potential risks. We have developed a practical approach to identifying risks, grounded in the policing and legal system of England and Wales. We identify 15 policing tasks that could be implemented using LLMs and 17 risks from their use, then illustrate with over 40 examples of impact on case progression. As good practice is agreed, many risks could be reduced. But this requires effort: we need to address these risks in a timely manner and define system wide impacts and benefits.

Evdoxia Taka, Debadyuti Bhattacharya, Joanne Garde-Hansen et al.

Recent advances in AI has made automated analysis of complex media content at scale possible while generating actionable insights regarding character representation along such dimensions as gender and age. Past works focused on quantifying representation from audio/video/text using AI models, but without having the audience in the loop. We ask, even if character distribution along demographic dimensions are available, how useful are those to the general public? Do they actually trust the numbers generated by AI models? Our work addresses these open questions by proposing a new AI-based character representation tool and performing a thorough user study. Our tool has two components: (i) An analytics extraction model based on the Contrastive Language Image Pretraining (CLIP) foundation model that analyzes visual screen data to quantify character representation across age and gender; (ii) A visualization component effectively designed for presenting the analytics to lay audience. The user study seeks empirical evidence on the usefulness and trustworthiness of the AI-generated results for carefully chosen movies presented in the form of our visualizations. We found that participants were able to understand the analytics in our visualizations, and deemed the tool `overall useful'. Participants also indicated a need for more detailed visualizations to include more demographic categories and contextual information of the characters. Participants' trust in AI-based gender and age models is seen to be moderate to low, although they were not against the use of AI in this context. Our tool including code, benchmarking, and the user study data can be found at https://github.com/debadyuti0510/Character-Representation-Media.

Evdoxia Taka, Yuri Nakao, Ryosuke Sonoda et al.

Fairness is a growing concern for high-risk decision-making using Artificial Intelligence (AI) but ensuring it through purely technical means is challenging: there is no universally accepted fairness measure, fairness is context-dependent, and there might be conflicting perspectives on what is considered fair. Thus, involving stakeholders, often without a background in AI or fairness, is a promising avenue. Research to directly involve stakeholders is in its infancy, and many questions remain on how to support stakeholders to feedback on fairness, and how this feedback can be integrated into AI models. Our work follows an approach where stakeholders can give feedback on specific decision instances and their outcomes with respect to their fairness, and then to retrain an AI model. In order to investigate this approach, we conducted two studies of a complex AI model for credit rating used in loan applications. In study 1, we collected feedback from 58 lay users on loan application decisions, and conducted offline experiments to investigate the effects on accuracy and fairness metrics. In study 2, we deepened this investigation by showing 66 participants the results of their feedback with respect to fairness, and then conducted further offline analyses. Our work contributes two datasets and associated code frameworks to bootstrap further research, highlights the opportunities and challenges of employing lay user feedback for improving AI fairness, and discusses practical implications for developing AI applications that more closely reflect stakeholder views about fairness.

Evdoxia Taka, Sebastian Stein, John H. Williamson

Despite growing interest in probabilistic modeling approaches and availability of learning tools, people with no or less statistical background feel hesitant to use them. There is need for tools for communicating probabilistic models to less experienced users more intuitively to help them build, validate, use effectively or trust probabilistic models. Users' comprehension of probabilistic models is vital in these cases and interactive visualizations could enhance it. Although there are various studies evaluating interactivity in Bayesian reasoning and available tools for visualizing the sample-based distributions, we focus specifically on evaluating the effect of interaction on users' comprehension of probabilistic models' structure. We conducted a user study based on our Interactive Pair Plot for visualizing models' distribution and conditioning the sample space graphically. Our results suggest that improvements in the understanding of the interaction group are most pronounced for more exotic structures, such as hierarchical models or unfamiliar parameterizations in comparison to the static group. As the detail of the inferred information increases, interaction does not lead to considerably longer response times. Finally, interaction improves users' confidence.