CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs
For researchers and practitioners in automated code generation and GPU programming, this benchmark highlights the persistent gap between current AI models and expert-level hardware-aware optimization.
The paper introduces CUDAHercules, a benchmark for evaluating automated CUDA programming against human-expert systems across multiple GPU architectures. Results show that even the strongest models (e.g., Claude-Opus-4.6) rarely match expert performance, indicating a large gap between runnable and optimized CUDA.
Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence.