Stochastic Code Generation
This addresses code generation challenges for developers, but the results are incremental as no significant improvement was achieved.
The study tackled the problem of large language models struggling with coherent long code generation by applying a latent stochastic process technique from text generation to code generation, but found that the modified model performed similarly to the baseline CodeParrot on the HumanEval benchmark.
Large language models pre-trained for code generation can generate high-quality short code but often struggle with generating coherent long code and understanding higher-level or system-level specifications. This issue is also observed in language modeling for long text generation, and one proposed solution is the use of a latent stochastic process. This approach involves generating a document plan and then producing text that is consistent with it. In this study, we investigate whether this technique can be applied to code generation to improve coherence. We base our proposed encoder and decoder on the pre-trained GPT-2 based CodeParrot model and utilize the APPS dataset for training. We evaluate our results using the HumanEval benchmark and observe that the modified Time Control model performs similarly to CodeParrot on this evaluation.