Demo-Craft: Using In-Context Learning to Improve Code Generation in Large Language Models
This work addresses code generation challenges for developers using LLMs, but it appears incremental as it builds on existing in-context learning methods with added latent concept learning.
The paper tackles the problem of generating executable code from natural language instructions in Large Language Models by proposing Demo-Craft, which uses in-context learning and latent concept learning, resulting in approximately 2x improvement in pass@k and 3x improvement in new metrics like correctness@k and similarity@k on MBPP and Humaneval datasets.
Generating executable code from natural language instructions using Large Language Models (LLMs) poses challenges such as semantic ambiguity and understanding taskspecific contexts. To address these issues, we propose a system called DemoCraft, which enhances code generation by leveraging in-context learning and demonstration selection, combined with latent concept learning. Latent concept learning introduces additional concept tokens, which are trainable embeddings that capture task-specific knowledge. We then test our system on two major datasets: MBPP and Humaneval. Our experimental results demonstrate that the proposed system achieves an approximate 2x increase in the pass@k metric compared to baseline models. Furthermore, we introduce two novel evaluation metrics: correctness@k and similarity@k. Our empirical studies indicate that our system attains nearly a 3x improvement in these metrics as well.