CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
This work addresses a gap in benchmarking for software engineering by providing a tool to assess LLMs' program semantic reasoning, which is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating large language models' (LLMs) code reasoning capabilities beyond surface-level patterns by introducing CORE, a benchmark with 12,553 static analysis tasks across multiple programming languages, and found that while LLMs perform well on dependency identification, they struggle with deeper semantic understanding and multi-step reasoning.
Large language models (LLMs) have been widely adopted across diverse domains of software engineering, such as code generation, program repair, and vulnerability detection. These applications require understanding beyond surface-level code patterns: value propagation, control flow, and interdependence between program elements. However, existing benchmarks primarily evaluate end-to-end outcomes, such as whether code is correctly repaired or generated, leaving the models' ability for program semantic reasoning underexplored. This work presents CORE, a high-quality, human-verified benchmark designed to evaluate LLMs on fundamental static analysis tasks. CORE includes 12,553 task instances spanning data dependency, control dependency, and information flow across programs written in C/C++, Java, and Python. To ensure semantic diversity and reasoning complexity, we propose a semantics-aware diverse sampling strategy that selects targets and task instances based on structural coverage and dependency depth. We evaluate 10 mainstream LLMs and show that, while they perform well at identifying dependencies, models still struggle with tasks that require deeper semantic understanding and multi-step reasoning. We further conduct qualitative analyses to uncover key challenges, such as complex control structures and backward dependency patterns, offering insights into improving LLMs' code reasoning capabilities.