Caitlin A. Stamatis

Ziyi Zhu, Olivier Tieleman, Caitlin A. Stamatis et al. · cambridge

Realistic user simulation is crucial for training and evaluating task-oriented dialogue (TOD) systems, yet creating simulators that accurately replicate human behavior remains challenging. A key property of effective simulators is their ability to expose failure modes of the systems they evaluate. We present an adversarial training framework that iteratively improves user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. Applied to mental health support chatbots, our approach demonstrates that fine-tuned simulators dramatically outperform zero-shot base models at surfacing system issues, and adversarial training further enhances diversity, distributional alignment, and predictive validity. The resulting simulator achieves a strong correlation between simulated and real failure occurrence rates across diverse chatbot configurations while maintaining low distributional divergence of failure modes. Discriminator accuracy decreases drastically after three adversarial iterations, suggesting improved realism. These results provide evidence that adversarial training is a promising approach for creating realistic user simulators in mental health support TOD domains, enabling rapid, reliable, and cost-effective system evaluation before deployment.

Caitlin A. Stamatis, Emma C. Wolfe, Matteo Malgaroli et al.

Background: Many people who could benefit from therapy do not receive it. Conversational AI is increasingly used for mental health support, yet it is unclear which barriers AI helps mitigate. We examined whether evaluation-sensitive (shame/stigma) and structural barriers (cost/coverage/access) to psychotherapy predict perceived helpfulness of an AI mental health conversational tool (Ash), and whether effects differ by prior therapy experience or user engagement. Methods: Participants (n=395) rated Ash's helpfulness (1-5) and described barriers to therapy. Open-text responses were coded for shame/stigma, access, and cost/coverage themes. Linear regressions examined associations between barriers and perceived helpfulness, adjusting for demographics and mental health, with moderation by therapy experience. Results: Shame/stigma (B=.45, p<.001) and access barriers (B=.31, p=.020) predicted higher perceived helpfulness but cost/coverage did not (B=.13, p=.262). Prior therapy experience moderated the shame effect (interaction B=.56, p=.036): shame predicted higher helpfulness among therapy-experienced users ($Δ$=.62, p<.001) but not therapy-naive users ($Δ$=.03, p=.877). Among therapy-experienced participants (n=258), shame/stigma (B=.75, p<.001) and access barriers (B=.51, p=.006) predicted rating Ash more favorably. Access barriers predicted higher engagement (IRR=1.64, p<.001) and cost/coverage barriers predicted 70% more sessions (IRR=1.70, p<.001). Shame/stigma was not associated with total sessions (IRR=.80, p=.094). Conclusions: AI mental health support was perceived as most helpful by users facing shame/stigma and access barriers, particularly for therapy-experienced individuals. Access and cost barriers were most predictive of usage intensity, suggesting unmet needs. Findings highlight the importance of aligning AI tools for emotional support with user-reported barriers.