Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
This addresses the need for more realistic and domain-relevant AI psychometrics, particularly for applications like law enforcement, though it is incremental as it builds on existing methods with new data and integration.
The paper tackles the problem of evaluating AI systems in roles requiring emotional and ethical judgment by proposing a framework that uses situational judgment tests and sophisticated personas, resulting in a dataset of 8,500 personas, 4,000 SJTs, and 300,000 responses for a law enforcement assistant case study.
AI psychometrics evaluates AI systems in roles that traditionally require emotional judgment and ethical consideration. Prior work often reuses human trait inventories (Big Five, \hexaco) or ad hoc personas, limiting behavioral realism and domain relevance. We propose a framework that (1) uses situational judgment tests (SJTs) from realistic scenarios to probe domain-specific competencies; (2) integrates industrial-organizational and personality psychology to design sophisticated personas which include behavioral and psychological descriptors, life history, and social and emotional functions; and (3) employs structured generation with population demographic priors and memoir inspired narratives, encoded with Pydantic schemas. In a law enforcement assistant case study, we construct a rich dataset of personas drawn across 8 persona archetypes and SJTs across 11 attributes, and analyze behaviors across subpopulation and scenario slices. The dataset spans 8,500 personas, 4,000 SJTs, and 300,000 responses. We will release the dataset and all code to the public.