PRMB: Benchmarking Reward Models in Long-Horizon CBT-based Counseling Dialogue
This work addresses the need for better assessment of reward models in mental health applications, though it is incremental as it focuses on benchmarking rather than developing new methods.
The authors tackled the problem of evaluating reward models in long-horizon cognitive behavioral therapy (CBT)-based counseling dialogues by introducing PRMB, a comprehensive benchmark spanning 6 sessions and 21 scenarios, which reveals generalization defects in state-of-the-art models and shows a positive correlation with downstream performance.
Large language models (LLMs) hold potential for mental healthcare applications, particularly in cognitive behavioral therapy (CBT)-based counseling, where reward models play a critical role in aligning LLMs with preferred therapeutic behaviors. However, existing reward model evaluations often fail to capture alignment effectiveness in long-horizon interventions due to limited coverage of process-oriented datasets and misalignment between evaluation targets and psychological alignment objectives. To address these limitations, we present PRMB, a comprehensive benchmark tailored for evaluating reward models in multi-session CBT counseling. PRMB spans 6 sessions and 21 diverse negative scenarios, incorporating both pairwise and Best-of-N preference evaluations. We demonstrate a positive correlation between our benchmark and downstream counseling dialogue performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art reward models, revealing their generalization defects that were not discovered by previous benchmarks and highlighting the potential of generative reward models. Furthermore, we delve into examining the effectiveness of inference-time strategy for the evaluation of reward models and analyzing the impact factors of generative reward models. This work advances intelligent informatics for personalized healthcare by establishing a framework for reward model assessment in mental health dialogues. Evaluation code and datasets are publicly available at https://github.com/YouKenChaw/PRMB