ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
For developers of LLM-based agents, this provides a more granular evaluation method to identify and improve specific reasoning deficiencies, though its impact is limited to the agent evaluation domain.
ClawTrack introduces a dual-assessment benchmark that evaluates LLM-based agents on both task outcomes and process quality, using 320 tasks and 12,541 rubric items. Testing 21 models over 16,000+ trials, it shows process scores can attribute failures to specific reasoning dimensions, with result verification being the main bottleneck, and that process-based filtering improves post-training performance.
As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.