ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents
This work addresses the problem of evaluating proactive dialogue capabilities for researchers and developers in AI, though it is incremental as it builds on existing domain-specific approaches.
The authors tackled the fragmented evaluation of proactive dialogue in large language models by proposing ProactiveEval, a unified framework that decomposes proactive dialogue into target planning and dialogue guidance, establishing metrics and generating data across 6 domains, and found that DeepSeek-R1 and Claude-3.7-Sonnet performed exceptionally in specific tasks.
Proactive dialogue has emerged as a critical and challenging research problem in advancing large language models (LLMs). Existing works predominantly focus on domain-specific or task-oriented scenarios, which leads to fragmented evaluations and limits the comprehensive exploration of models' proactive conversation abilities. In this work, we propose ProactiveEval, a unified framework designed for evaluating proactive dialogue capabilities of LLMs. This framework decomposes proactive dialogue into target planning and dialogue guidance, establishing evaluation metrics across various domains. Moreover, it also enables the automatic generation of diverse and challenging evaluation data. Based on the proposed framework, we develop 328 evaluation environments spanning 6 distinct domains. Through experiments with 22 different types of LLMs, we show that DeepSeek-R1 and Claude-3.7-Sonnet exhibit exceptional performance on target planning and dialogue guidance tasks, respectively. Finally, we investigate how reasoning capabilities influence proactive behaviors and discuss their implications for future model development.