Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
For researchers and developers of interactive coding assistants, this work provides a benchmark and method to evaluate and improve the understudied dialogue dimension of coding agents.
The authors introduce Dialogue SWE-Bench, a benchmark for evaluating coding agents through dialogue with a simulated user, and propose a schema-guided agent that improves dialogue performance by 3-14% over baselines, finding that coding ability does not correlate with dialogue ability.
AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.