Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code
This work addresses the lack of high-quality datasets and accurate reward functions for training reinforcement learning models to provide helpful natural language feedback on erroneous code, benefiting developers and educators using code LLMs.
This paper introduces Coffee-Gym, an RL environment designed to train models that provide feedback on code editing. It includes a dataset of human code edit traces and machine-written feedback, along with a reward function, CoffeeEval, that assesses feedback helpfulness by evaluating revised code against unit tests. Using Coffee-Gym, the authors developed feedback models that improved open-source code LLMs' editing capabilities to a level comparable with closed-source LLMs.
This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans' code edit traces for coding questions and machine-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs' code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available.