Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning
For practitioners training code-generating LLMs, this work offers a more efficient post-training alternative to online RL, though the gains are incremental over existing methods.
This paper explores offline reinforcement learning for post-training LLMs on code generation, showing it improves performance, especially for small models and hard problems, while avoiding costly online inference and verification.
Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models. However, online RL for code generation involves LLM inference and verification of the generated output, which can take considerable time and resources. In this paper, we explore the application of offline RL to code-generating models by leveraging existing code datasets. Our experiments demonstrate that offline RL is an effective training strategy for improving LLM performance. We show that offline RL can be especially beneficial for small LLMs and challenging coding problems.