Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors
This work addresses the lack of scalable feedback mechanisms for training peer counselors to support mental health issues, though it is incremental as it builds on existing LLM methods with domain-specific adaptations.
The paper tackles the problem of providing detailed feedback to novice peer counselors by leveraging large language models to generate contextualized, multi-level feedback, demonstrating through expert evaluation that their method minimizes harmful and low-quality feedback in high-stakes scenarios.
Realistic practice and tailored feedback are key processes for training peer counselors with clinical skills. However, existing mechanisms of providing feedback largely rely on human supervision. Peer counselors often lack mechanisms to receive detailed feedback from experienced mentors, making it difficult for them to support the large number of people with mental health issues who use peer counseling. Our work aims to leverage large language models to provide contextualized and multi-level feedback to empower peer counselors, especially novices, at scale. To achieve this, we co-design with a group of senior psychotherapy supervisors to develop a multi-level feedback taxonomy, and then construct a publicly available dataset with comprehensive feedback annotations of 400 emotional support conversations. We further design a self-improvement method on top of large language models to enhance the automatic generation of feedback. Via qualitative and quantitative evaluation with domain experts, we demonstrate that our method minimizes the risk of potentially harmful and low-quality feedback generation which is desirable in such high-stakes scenarios.