A Contrastive Framework for Neural Text Generation
This addresses the issue of generating coherent and diverse text for natural language processing applications, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackled the problem of degenerate text generation from neural language models, which often produces unnatural and repetitive outputs, by proposing a contrastive framework that calibrates token representations and uses a decoding method to enhance diversity and coherence. The result is a significant outperformance over state-of-the-art methods on three benchmarks across two languages, as evaluated by human and automatic metrics.
Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.