Adversarial Training for Failure-Sensitive User Simulation in Mental Health Dialogue Optimization
This work addresses the problem of reliable and cost-effective evaluation for mental health support chatbots before deployment, though it is incremental as it builds on existing adversarial training methods applied to a specific domain.
The paper tackled the challenge of creating realistic user simulators for task-oriented dialogue systems, particularly in mental health support chatbots, by introducing an adversarial training framework that iteratively improves simulator realism; the resulting simulator achieved a strong correlation between simulated and real failure occurrence rates and reduced discriminator accuracy after three adversarial iterations, indicating enhanced realism.
Realistic user simulation is crucial for training and evaluating task-oriented dialogue (TOD) systems, yet creating simulators that accurately replicate human behavior remains challenging. A key property of effective simulators is their ability to expose failure modes of the systems they evaluate. We present an adversarial training framework that iteratively improves user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. Applied to mental health support chatbots, our approach demonstrates that fine-tuned simulators dramatically outperform zero-shot base models at surfacing system issues, and adversarial training further enhances diversity, distributional alignment, and predictive validity. The resulting simulator achieves a strong correlation between simulated and real failure occurrence rates across diverse chatbot configurations while maintaining low distributional divergence of failure modes. Discriminator accuracy decreases drastically after three adversarial iterations, suggesting improved realism. These results provide evidence that adversarial training is a promising approach for creating realistic user simulators in mental health support TOD domains, enabling rapid, reliable, and cost-effective system evaluation before deployment.