Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students
This addresses the problem of generating realistic synthetic classroom data for education tool developers, though it is incremental as it builds on prior analyses of student bugs.
The study compared bug distributions in code generated by large language models (LLMs) versus computing students, finding that unguided LLMs produce unrealistic errors, but with guidance on common errors and frequencies, they can generate plausible synthetic bug patterns.
Large language models (LLMs) present an exciting opportunity for generating synthetic classroom data. Such data could include code containing a typical distribution of errors, simulated student behaviour to address the cold start problem when developing education tools, and synthetic user data when access to authentic data is restricted due to privacy reasons. In this research paper, we conduct a comparative study examining the distribution of bugs generated by LLMs in contrast to those produced by computing students. Leveraging data from two previous large-scale analyses of student-generated bugs, we investigate whether LLMs can be coaxed to exhibit bug patterns that are similar to authentic student bugs when prompted to inject errors into code. The results suggest that unguided, LLMs do not generate plausible error distributions, and many of the generated errors are unlikely to be generated by real students. However, with guidance including descriptions of common errors and typical frequencies, LLMs can be shepherded to generate realistic distributions of errors in synthetic code.