Generating Sequences by Learning to Self-Correct
This addresses the issue of constraint violations in sequence generation for applications like programming and content moderation, offering a method to enhance off-the-shelf models without retraining, though it is incremental as it builds on existing generation techniques.
The paper tackles the problem of language models frequently violating semantic constraints in sequence generation tasks by introducing Self-Correction, an approach that uses a separate corrector to iteratively revise outputs, resulting in improvements across mathematical program synthesis, lexically-constrained generation, and toxicity control tasks.
Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs. Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation. We present Self-Correction, an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations. To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations. We show that Self-Correction improves upon the base generator in three diverse generation tasks - mathematical program synthesis, lexically-constrained generation, and toxicity control - even when the corrector is much smaller than the base generator.