Thomas D. Hull

Ziyi Zhu, Olivier Tieleman, Caitlin A. Stamatis et al. · cambridge

Realistic user simulation is crucial for training and evaluating task-oriented dialogue (TOD) systems, yet creating simulators that accurately replicate human behavior remains challenging. A key property of effective simulators is their ability to expose failure modes of the systems they evaluate. We present an adversarial training framework that iteratively improves user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. Applied to mental health support chatbots, our approach demonstrates that fine-tuned simulators dramatically outperform zero-shot base models at surfacing system issues, and adversarial training further enhances diversity, distributional alignment, and predictive validity. The resulting simulator achieves a strong correlation between simulated and real failure occurrence rates across diverse chatbot configurations while maintaining low distributional divergence of failure modes. Discriminator accuracy decreases drastically after three adversarial iterations, suggesting improved realism. These results provide evidence that adversarial training is a promising approach for creating realistic user simulators in mental health support TOD domains, enabling rapid, reliable, and cost-effective system evaluation before deployment.

Xinyang Ren, Hannah A Burkhardt, Patricia A Areán et al.

Prior work has shown that analyzing the use of first-person singular pronouns can provide insight into individuals' mental status, especially depression symptom severity. These findings were generated by counting frequencies of first-person singular pronouns in text data. However, counting doesn't capture how these pronouns are used. Recent advances in neural language modeling have leveraged methods generating contextual embeddings. In this study, we sought to utilize the embeddings of first-person pronouns obtained from contextualized language representation models to capture ways these pronouns are used, to analyze mental status. De-identified text messages sent during online psychotherapy with weekly assessment of depression severity were used for evaluation. Results indicate the advantage of contextualized first-person pronoun embeddings over standard classification token embeddings and frequency-based pronoun analysis results in predicting depression symptom severity. This suggests contextual representations of first-person pronouns can enhance the predictive utility of language used by people with depression symptoms.

Benjamin Brindle, George A. Bonanno, Thomas Derrick Hull et al.

Predicting treatment non-response for anxiety and depression is challenging, in part because of sparse symptom assessments in real-world care. We examined whether passively captured, fine-grained emotions serve as linguistic markers of treatment outcomes by analyzing 12 weeks of de-identified teletherapy transcripts from 12,043 U.S. patients with moderate-to-severe anxiety and depression symptoms. A transformer-based small language model extracted patients' emotions at the talk-turn level; a state-space model (VISTA-SSM) clustered subgroups based on emotion dynamics over time and produced temporal networks. Two groups emerged: an improving group (n=8,230) and a non-response group (n=3,813) showing increased odds of symptom deterioration, and lower likelihood of clinically significant improvement. Temporal networks indicated that sadness and fear exerted most influence on emotion dynamics in non-responders, whereas improving patients showed balanced joy, sadness, and neutral expressions. Findings suggest that linguistic markers of emotional inflexibility can serve as scalable, interpretable, and theoretically grounded indicators for treatment risk stratification.