UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
This work addresses a critical issue for researchers and developers evaluating LLM-based coding agents by improving benchmark reliability, though it is incremental as it builds on existing methods for test generation.
The paper tackled the problem of insufficient test cases in the SWE-Bench benchmark for coding agents, which allowed erroneous patches to pass, and introduced UTBoost, a framework that generated test cases to correct 345 mislabeled patches, affecting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries and causing ranking changes.
The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities of these agents, SWE-Bench uses real-world problems based on GitHub issues and their corresponding pull requests. However, the manually written test cases included in these pull requests are often insufficient, allowing generated patches to pass the tests without resolving the underlying issue. To address this challenge, we introduce UTGenerator, an LLM-driven test case generator that automatically analyzes codebases and dependencies to generate test cases for real-world Python projects. Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation. In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench. These corrections, impacting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries, yield 18 and 11 ranking changes, respectively.