Towards Evaluation of Implicit Software World Models in Coding LLMs
For AI software engineering, this work highlights a critical gap in LLMs' understanding of runtime behavior, but the evaluation is preliminary and limited to a single benchmark.
This paper proposes evaluating coding LLMs on predicting execution resources (peak memory, wall-clock time, profiler outputs) beyond control flow, using SWE-bench Verified. All tested models show modest performance and brittle behavior, indicating a lack of understanding of software execution.
Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.