LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
For researchers and practitioners working on automated compiler issue resolution, this paper provides a benchmark and reveals limitations of current LLMs, with an ensemble method offering modest improvements.
The paper introduces LLVM-Bench, the first large-scale benchmark for LLVM compiler issue resolution with 423 tasks, and LLVM-Gym, an automated evaluation platform. Current LLM-based methods achieve limited success, but an ensemble approach (LLVM-Ens) improves resolution rate to 21.99%.
LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.