Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
For researchers evaluating LLM code reasoning, this work provides a more realistic benchmark and reveals that current evaluations may overestimate LLM performance due to simplistic problem sets.
The paper constructs a dataset of 1200 code reasoning problems with realistic complexities (e.g., inter-procedural dependencies, complex types) and categorizes them into lower/higher complexity using nine metrics. It finds that existing benchmarks mostly contain lower-complexity problems, limiting generalizability assessments of LLMs.
Code reasoning tasks are becoming prevalent in large language model (LLM) assessments. Yet, there is a dearth of studies on the impact of real-world complexities on code reasoning, e.g., inter- or intra-procedural dependencies, API calls, deeply nested constructs, and non-primitive complex types. Evaluating LLMs under such a simplistic setting poses a significant threat to assumptions about their generalizability in practice. To enable a more realistic evaluation of code reasoning, we construct a dataset of 1200 reasoning problems from two sources: existing code reasoning benchmarks and popular GitHub Python repositories. Our pipeline leverages static and dynamic program analysis to automatically serialize/deserialize compound, complex, and custom types galore in real-world code, going far beyond only primitive types used in prior studies. A key feature of our dataset is categorizing each reasoning problem as Lower Complexity (LC) or Higher Complexity (HC) via a principled majority-vote mechanism over nine diverse and interpretable code-complexity metrics, yielding two well-separated, semantically meaningful categories of problem difficulty suitable for precise calibration of LLM reasoning ability. This categorization shows that the problems used in existing code-reasoning evaluation mostly belong to the LC category, failing to represent real-world complexity.