Learning to Write with Coherence From Negative Examples
This work addresses coherence issues in natural language generation for applications like text continuation, though it appears incremental as it builds on existing contrastive training methods.
The paper tackled the problem of improving coherence in neural text generation by proposing a writing relevance training method that uses negative examples to contrast context and generated sentences with positive continuations. The result showed that this approach outperformed unlikelihood training in human evaluations on a commonsense NLI corpus, demonstrating efficacy in enhancing coherence.
Coherence is one of the critical factors that determine the quality of writing. We propose writing relevance (WR) training method for neural encoder-decoder natural language generation (NLG) models which improves coherence of the continuation by leveraging negative examples. WR loss regresses the vector representation of the context and generated sentence toward positive continuation by contrasting it with the negatives. We compare our approach with Unlikelihood (UL) training in a text continuation task on commonsense natural language inference (NLI) corpora to show which method better models the coherence by avoiding unlikely continuations. The preference of our approach in human evaluation shows the efficacy of our method in improving coherence.