StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code
This addresses the need for benchmarks reflecting non-expert users in code generation, with implications for teaching and deployment, though it is incremental as it builds on existing evaluation frameworks.
The authors introduced StudentEval, a benchmark of 1,749 prompts written by 80 beginner programmers for 48 coding problems, to evaluate code LLMs, finding it better discriminates model performance than existing benchmarks and revealing mixed success rates and misleading effects of nondeterministic sampling.
Code LLMs are being rapidly deployed and there is evidence that they can make professional programmers more productive. Current benchmarks for code generation measure whether models generate correct programs given an expert prompt. In this paper, we present a new benchmark containing multiple prompts per problem, written by a specific population of non-expert prompters: beginning programmers. StudentEval contains 1,749 prompts for 48 problems, written by 80 students who have only completed one semester of Python programming. Our students wrote these prompts while working interactively with a Code LLM, and we observed very mixed success rates. We use StudentEval to evaluate 5 Code LLMs and find that StudentEval is a better discriminator of model performance than existing benchmarks. We analyze the prompts and find significant variation in students' prompting techniques. We also find that nondeterministic LLM sampling could mislead students into thinking that their prompts are more (or less) effective than they actually are, which has implications for how to teach with Code LLMs.