TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
This addresses a critical but overlooked task in automated software engineering for software engineers, though it is incremental as it focuses on benchmarking rather than solving the migration problem directly.
The authors tackled the problem of evaluating software migration capabilities in real-world Python projects by introducing TimeMachine-bench, a benchmark based on GitHub repositories with failing tests due to dependency updates, and found that LLMs show promise but face substantial reliability challenges like spurious solutions and unnecessary edits.
With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a benchmark designed to evaluate software migration in real-world Python projects. Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.