Coarse-Tuning Models of Code with Reinforcement Learning Feedback
This addresses the issue of unreliable code synthesis from LLMs for developers, though it is an incremental improvement over existing methods.
The paper tackles the problem of improving code generation in large language models (LLMs) by proposing RLCF, a reinforcement learning method that uses compiler and LLM feedback to train pre-trained models, resulting in increased compilation, execution, and correctness rates, often matching the performance of models 2x-8x larger.
Large Language Models (LLMs) pre-trained on code have recently emerged as the dominant approach to program synthesis. However, these models are trained using next-token prediction, which ignores the syntax and semantics of code. We propose RLCF, that further trains a pre-trained LLM via reinforcement learning, using feedback from a grounding function that scores the quality of the code. The grounding function uses (i) compiler-derived feedback on whether the code it generates passes a set of correctness checks; and (ii) feedback from a different LLM that compares the generated code to a reference code. RLCF is model- and language-agnostic. We empirically evaluate it on the MBJP and MathQA tasks for Java. Our experiments show that RLCF raises the odds that an LLM-generated program compiles, is executable, and produces the right output on tests, often allowing LLMs to match the performance of 2x-8x larger LLMs.