Improving Constrained Generation in Language Models via Self-Distilled Twisted Sequential Monte Carlo
This addresses constrained generation for NLP applications, but it is incremental as it builds on existing twisted Sequential Monte Carlo methods.
The paper tackled the problem of constrained text generation in language models, where learning is difficult due to sparse rewards, by using self-distillation to iteratively refine the base model, resulting in substantial gains in generation quality.
Recent work has framed constrained text generation with autoregressive language models as a probabilistic inference problem. Among these, Zhao et al. (2024) introduced a promising approach based on twisted Sequential Monte Carlo, which incorporates learned twist functions and twist-induced proposals to guide the generation process. However, in constrained generation settings where the target distribution concentrates on outputs that are unlikely under the base model, learning becomes challenging due to sparse and uninformative reward signals. We show that iteratively refining the base model through self-distillation alleviates this issue by making the model progressively more aligned with the target, leading to substantial gains in generation quality.