PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency
This work addresses the need for systematic evaluation of persona agents before trusting them as substitutes for human participants, representing an incremental advancement in evaluation methodology.
The authors tackled the problem of verifying the consistency of persona agents based on large language models by proposing PICon, a multi-turn interrogation framework that evaluates internal, external, and retest consistency, finding that even highly consistent systems fail to meet human baselines across all dimensions.
Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants across diverse domains. Yet there is no systematic method for verifying whether a persona agent's responses remain free of contradictions and factual inaccuracies throughout an interaction. A principle from interrogation methodology offers a lens: no matter how elaborate a fabricated identity, systematic interrogation will expose its contradictions. We apply this principle to propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning. PICon evaluates consistency along three core dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). Evaluating seven groups of persona agents alongside 63 real human participants, we find that even systems previously reported as highly consistent fail to meet the human baseline across all three dimensions, revealing contradictions and evasive responses under chained questioning. This work provides both a conceptual foundation and a practical methodology for evaluating persona agents before trusting them as substitutes for human participants. We provide the source code and an interactive demo at: https://kaist-edlab.github.io/picon/