Prompt Variability Effects On LLM Code Generation
This addresses the need for robust evaluation methods in code generation for developers and researchers, but it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of LLM code generation quality being sensitive to prompt variations, particularly user background, by proposing a synthetic evaluation pipeline and persona-based approach to quantify this sensitivity, showing their utility through experiments.
Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user's background and familiarity with software development. It is therefore important to quantify LLM's sensitivity to variations in the input. To this end we propose a synthetic evaluation pipeline for code generation with LLMs, as well as a systematic persona-based evaluation approach to expose qualitative differences of LLM responses dependent on prospective user background. Both proposed methods are completely independent from specific programming tasks and LLMs, and thus are widely applicable. We provide experimental evidence illustrating utility of our methods and share our code for the benefit of the community.