LLM Personas as a Substitute for Field Experiments in Method Benchmarking
This work addresses the bottleneck of rapid methodological progress in societal systems by offering a synthetic benchmarking approach, though it is incremental as it builds on existing LLM persona simulation concepts with new theoretical guarantees.
The paper tackles the problem of costly field experiments for benchmarking methods in societal systems by proposing LLM-based persona simulation as a cheap alternative, proving that under specific conditions (aggregate-only observation and method-blind evaluation), swapping humans for personas is indistinguishable from changing the evaluation population, and providing explicit bounds on the number of persona evaluations needed to reliably distinguish methods.
Field experiments (A/B tests) are often the most credible benchmark for methods (algorithms) in societal systems, but their cost and latency bottleneck rapid methodological progress. LLM-based persona simulation offers a cheap synthetic alternative, yet it is unclear whether replacing humans with personas preserves the benchmark interface that adaptive methods optimize against. We prove an if-and-only-if characterization: when (i) methods observe only the aggregate outcome (aggregate-only observation) and (ii) evaluation depends only on the submitted artifact and not on the method's identity or provenance (method-blind evaluation), swapping humans for personas is just panel change from the method's point of view, indistinguishable from changing the evaluation population (e.g., New York to Jakarta). Furthermore, we move from validity to usefulness: we define an information-theoretic discriminability of the induced aggregate channel and show that making persona benchmarking as decision-relevant as a field experiment is fundamentally a sample-size question, yielding explicit bounds on the number of independent persona evaluations required to reliably distinguish meaningfully different methods at a chosen resolution.