On Leakage of Code Generation Evaluation Datasets
This addresses data integrity issues for researchers and developers evaluating code generation models, though it is incremental as it focuses on a specific domain.
The paper tackles the problem of contamination in code generation evaluation datasets for large language models, identifying three sources of leakage and releasing a new uncontaminated benchmark called Less Basic Python Problems (LBPP) with 161 prompts and solutions.
In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models. We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection. To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp .