ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance
For researchers and developers of LM-based coding agents, this benchmark highlights the challenge of maintaining context across multiple fixes, which is overlooked by existing single-bug benchmarks.
ChainSWE introduces a benchmark for evaluating LM agents on sequential, dependent bug fixes in a shared codebase, revealing a performance drop of up to 70% as chain length increases.
Language model (LM) agents are increasingly deployed to maintain codebases over extended periods, fixing streams of related defects while carrying context from one fix to the next. Yet existing software engineering (SWE) benchmarks evaluate models one bug at a time: the repository is reset, the codebase is re-read, and a single self-contained issue is graded in isolation. This setting collapses a continuous maintenance workflow into a series of independent sessions, ignoring the cumulative dependencies that make real-world bug fixing challenging. To bridge this gap, we introduce ChainSWE, the first benchmark for evaluating agents on sequential, dependent bug fixes within a shared codebase. We collect chronological chains of 304 issues across 54 Python projects, mined from six SWE-bench-family datasets. Our evaluation across a range of agents and models reveals a consistent performance drop by up to 70% as the chain length increases.