Look-back Decoding for Open-Ended Text Generation
This addresses the issue of topic drift and repetition in text generation for applications like story writing, though it is incremental as it builds on existing decoding methods.
The paper tackled the problem of generating coherent and non-repetitive text in open-ended generation by proposing Look-back, a decoding algorithm that uses Kullback-Leibler divergence to monitor distribution shifts, resulting in significantly better performance in automatic and human evaluations for document continuation and story generation.
Given a prefix (context), open-ended generation aims to decode texts that are coherent, which do not abruptly drift from previous topics, and informative, which do not suffer from undesired repetitions. In this paper, we propose Look-back, an improved decoding algorithm that leverages the Kullback-Leibler divergence to track the distribution distance between current and historical decoding steps. Thus Look-back can automatically predict potential repetitive phrase and topic drift, and remove tokens that may cause the failure modes, restricting the next token probability distribution within a plausible distance to the history. We perform decoding experiments on document continuation and story generation, and demonstrate that Look-back is able to generate more fluent and coherent text, outperforming other strong decoding methods significantly in both automatic and human evaluations.