Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes
This work addresses a gap in evaluating code language models by emphasizing diversity, which is crucial for developers and researchers in AI and software engineering, though it is incremental as it builds on existing evaluation methods.
The paper highlights the importance of evaluating code diversity in language models for code generation, proposing a systematic approach with metrics for similarity and functional correctness, and finds that current models produce functionally correct code with limited diversity, showing a positive correlation between test pass scores and inter-code similarity scores.
Language models (LMs) have exhibited impressive abilities in generating codes from natural language requirements. In this work, we highlight the diversity of code generated by LMs as a critical criterion for evaluating their code generation capabilities, in addition to functional correctness. Despite its practical implications, there is a lack of studies focused on assessing the diversity of generated code, which overlooks its importance in the development of code LMs. We propose a systematic approach to evaluate the diversity of generated code, utilizing various metrics for inter-code similarity as well as functional correctness. Specifically, we introduce a pairwise code similarity measure that leverages large LMs' capabilities in code understanding and reasoning, demonstrating the highest correlation with human judgment. We extensively investigate the impact of various factors on the quality of generated code, including model sizes, temperatures, training approaches, prompting strategies, and the difficulty of input problems. Our consistent observation of a positive correlation between the test pass score and the inter-code similarity score indicates that current LMs tend to produce functionally correct code with limited diversity.