Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering
This addresses the need for reliable evaluation of LLMs in software engineering, though it is incremental as it builds on existing survey and workshop insights to propose a new infrastructure.
The paper tackles the problem of evaluating large language models for code, which currently relies on narrow benchmarks with inconsistent practices, by introducing BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation to enable fair and realistic assessment.
Large language models for code are advancing fast, yet our ability to evaluate them lags behind. Current benchmarks focus on narrow tasks and single metrics, which hide critical gaps in robustness, interpretability, fairness, efficiency, and real-world usability. They also suffer from inconsistent data engineering practices, limited software engineering context, and widespread contamination issues. To understand these problems and chart a path forward, we combined an in-depth survey of existing benchmarks with insights gathered from a dedicated community workshop. We identified three core barriers to reliable evaluation: the absence of software-engineering-rich datasets, overreliance on ML-centric metrics, and the lack of standardized, reproducible data pipelines. Building on these findings, we introduce BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation. BEHELM provides a structured way to assess models across tasks, languages, input and output granularities, and key quality dimensions. Our goal is to reduce the overhead currently required to construct benchmarks while enabling a fair, realistic, and future-proof assessment of LLMs in software engineering.