Critic-Guided Decoding for Controlled Text Generation
This work addresses the challenge of controlled text generation for users needing precise language model outputs, representing an incremental improvement by hybridizing existing methods.
The authors tackled the problem of steering language model generation towards desired objectives and away from undesired content by proposing CriticControl, a critic decoding method that combines reinforcement learning and weighted decoding, resulting in more coherent and well-controlled texts across tasks like topic control, sentiment control, and detoxification, with superior generalization in zero-shot settings.
Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. In this work, we propose a novel critic decoding method for controlled language generation (CriticControl) that combines the strengths of reinforcement learning and weighted decoding. Specifically, we adopt the actor-critic framework to train an LM-steering critic from non-differentiable reward models. And similar to weighted decoding, our method freezes the language model and manipulates the output token distribution using called critic, improving training efficiency and stability. Evaluation of our method on three controlled generation tasks, namely topic control, sentiment control, and detoxification, shows that our approach generates more coherent and well-controlled texts than previous methods. In addition, CriticControl demonstrates superior generalization ability in zero-shot settings. Human evaluation studies also corroborate our findings.