Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning
This addresses the challenge of making language models safer and more controllable for users, though it is incremental as it builds on existing contrastive learning approaches.
The paper tackles the problem of controlling language models to avoid generating texts with undesirable attributes like toxicity and repetition, by introducing Click, a method that uses sequence likelihood contrastive learning to reduce generation probability of negative samples, achieving superior performance on tasks such as language detoxification, sentiment steering, and repetition reduction compared to strong baselines.
It has always been an important yet challenging problem to control language models to avoid generating texts with undesirable attributes, such as toxic language and unnatural repetition. We introduce Click for controllable text generation, which needs no modification to the model architecture and facilitates out-of-the-box use of trained models. It employs a contrastive loss on sequence likelihood, which fundamentally decreases the generation probability of negative samples (i.e., generations with undesirable attributes). It also adopts a novel likelihood ranking-based strategy to construct contrastive samples from model generations. On the tasks of language detoxification, sentiment steering, and repetition reduction, we show that Click outperforms strong baselines of controllable text generation and demonstrate the superiority of Click's sample construction strategy.