RubberDuckBench: A Benchmark for AI Coding Assistants
Provides a new benchmark for evaluating AI coding assistants on realistic, contextualized code questions, highlighting that current models are unreliable.
The paper introduces RubberDuckBench, a multilingual benchmark of real-world contextualized questions about code derived from GitHub pull request comments, and evaluates 20 LLMs on it. The best models (Grok 4, Claude Opus 4, GPT-5) achieve around 68-69% accuracy but fail to give consistent correct responses, with most points from partial credit and frequent hallucinations (58.3% of responses).
Programmers are turning to AI coding assistants to answer questions about their code. Benchmarks are needed to soundly evaluate these systems and understand their performance. To enable such a study, we curate a benchmark of real-world contextualized questions derived from Github pull request comments. Out of this work, we present RubberDuckBench: a multilingual benchmark of questions about code, along with detailed rubrics for evaluating answers. We evaluate a diverse set of 20 LLMs (proprietary & open-source) on answering these questions. We find that even state of the art models fail to give consistent, correct responses across the benchmark. Grok 4 (69.29%), Claude Opus 4 (68.5%), and GPT-5 (67.8%) perform best overall, but do not exhibit pairwise significant superiority over the next 9 best performing models. Most models obtain points through partial credit, with the best performing models only answering at most 2 questions completely correctly across all trials. Furthermore, models often hallucinate with lies in 58.3\% of responses on average. Cost analysis reveals no correlation between expense (API pricing or parameter count) and performance. We intend this benchmark to be a target for future research in trustworthy and correct AI coding assistants.