Teaching Models to Improve on Tape
This work addresses the challenge of improving LLMs' ability to generate content under specific constraints, which is incremental as it builds on existing feedback-based methods by integrating RL and meta-learning.
The paper tackles the problem of large language models struggling with constrained content generation by introducing CORGI, a reinforcement learning framework that uses simulated interaction sessions and corrective feedback to train models, resulting in consistent outperformance over baseline RL methods and improved generalization to new tasks.
Large Language Models (LLMs) often struggle when prompted to generate content under specific constraints. However, in such cases it is often easy to check whether these constraints are satisfied or violated. Recent works have shown that LLMs can benefit from such "corrective feedback". Here we claim that this skill of LLMs can be significantly enhanced via training. We introduce an RL framework for teaching models to use such rewards, by simulating interaction sessions, and rewarding the model according to its ability to satisfy the constraints. We refer to our method as CORGI (Controlled Generation with RL for Guided Interaction), and evaluate it on a variety of controlled generation tasks using unlabeled training data. We find that CORGI consistently outperforms the baseline reinforcement learning method that does not incorporate conversational feedback. Furthermore, CORGI's interactive framework enables meta-learning, allowing the LLM to generalize better to guided interaction in new tasks. Our results clearly show that conversational optimization, when combined with reinforcement learning, significantly improves the effectiveness of LLMs in controlled generation contexts.