Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
For researchers using user simulators for AI assistant training and evaluation, this work provides a systematic measurement of simulator fidelity and a method to improve it.
The paper introduces a method to measure the distributional gap between real and simulated user behaviors, evaluates 24 LLM-based user simulators on coding and writing tasks, and finds a large gap that varies across models. Combining complementary simulators reduces this gap.
As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behaviors of real users. While existing works train user simulators to generate human-like responses, whether they capture the broad and heterogeneous distribution of real user behaviors remains an open question. In this work, we introduce a method to measure the distributional gap between real and simulated user behaviors, validated through a human study and ablations. Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation, quantizes them into discrete distributions via clustering, then computes divergence metrics. We provide the first systematic evaluation of 24 LLM-based user simulators on coding and writing tasks, and reveal a large distributional gap from real users that varies across model families, scales, and behavioral facets. Pairwise comparisons show that most simulators behave similarly, while a few stand apart. Combining behaviorally complementary simulators brings the resulting distribution closer to real users compared to either simulator on its own. Finally, a TF-IDF analysis of the clusters surfaces interpretable patterns of behaviors that simulators capture, miss, and hallucinate.