What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents
For developers and evaluators of coding agents, this work provides a diagnostic tool to uncover process-level insights hidden by binary pass/fail metrics, enabling better failure analysis and run prioritization.
Coding agents are typically evaluated by resolve rate (pass/fail on tests), which ignores process differences. TraceProbe, a trajectory-diagnostic framework, normalizes runs into a nine-type action taxonomy and applies rule-based modules to detect anti-patterns and compare trajectories. Analyzing 2,500 trajectories from SWE-Bench Verified, it reveals that function selection and completion behavior localize success/failure, anti-patterns serve as difficulty clues, and resolved runs differ in efficiency and failed work.
Coding agents are ranked almost entirely by resolve rate: whether their final patch passes the target tests. Yet two agents can reach the same outcome through very different processes, and a single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens. This process evidence lives in the trajectory, which records a run's searches, reads, edits, tool calls, validation, and reversions. However, raw traces are heterogeneous and hard to compare across runs. We present TraceProbe, a trajectory-diagnostic framework that recovers what resolve rate hides. TraceProbe normalizes each raw run into a canonical nine-type action taxonomy with deterministic effect labels, then applies two rule-based modules: Insight names single-trajectory anti-patterns adapted from established debugging practice (e.g., search loops, verification skips), while Converge aligns pairs of runs and classifies where their behavior diverges under controlled references. Applying TraceProbe to 2,500 trajectories from five production settings on SWE-Bench Verified, we find that (i) file choice is too coarse to separate success from failure, whereas function selection and completion behavior localize it; (ii) Insight anti-patterns act mainly as corpus-level difficulty clues, with search loops the most stable; and (iii) even resolved runs differ in how quickly they reach relevant code and how much failed work they incur. Trajectory structure thus adds auditable diagnostic context to outcomes by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review.