Generating Unseen Code Tests In Infinitum
This addresses the issue of reliable LLM evaluation for developers and researchers, though it is incremental as it builds on existing benchmarking practices.
The paper tackles the problem of benchmark data leakage in evaluating Large Language Models (LLMs) for coding tasks by presenting a method to generate ongoing variations of benchmarks, and it implements an auto-regression benchmark for Python text-to-code generation to aid in debugging and tracking model changes.
Large Language Models (LLMs) are used for many tasks, including those related to coding. An important aspect of being able to utilize LLMs is the ability to assess their fitness for specific usages. The common practice is to evaluate LLMs against a set of benchmarks. While benchmarks provide a sound foundation for evaluation and comparison of alternatives, they suffer from the well-known weakness of leaking into the training data \cite{Xu2024Benchmarking}. We present a method for creating benchmark variations that generalize across coding tasks and programming languages, and may also be applied to in-house code bases. Our approach enables ongoing generation of test-data thus mitigating the leaking into the training data issue. We implement one benchmark, called \textit{auto-regression}, for the task of text-to-code generation in Python. Auto-regression is specifically created to aid in debugging and in tracking model generation changes as part of the LLM regression testing process.