Momentum Decoding: Open-ended Text Generation As Graph Exploration
This addresses a core problem in natural language processing for users of language models by providing a faster and efficient decoding method, though it is incremental as it builds on existing graph-based perspectives.
The paper tackles the degeneration problem in open-ended text generation with autoregressive language models, where maximization-based decoding leads to unnatural and repetitive text, by proposing momentum decoding, which views generation as graph exploration and encourages greedy exploration of new nodes while allowing returns with downgraded momentum, resulting in performance comparable to state-of-the-art methods with notably improved inference speed and computation FLOPs.
Open-ended text generation with autoregressive language models (LMs) is one of the core tasks in natural language processing. However, maximization-based decoding methods (e.g., greedy/beam search) often lead to the degeneration problem, i.e., the generated text is unnatural and contains undesirable repetitions. Existing solutions to this problem either introduce randomness prone to incoherence or require a look-ahead mechanism that demands extra computational overhead. In this study, we formulate open-ended text generation from a new perspective, i.e., we view it as an exploration process within a directed graph. Thereby, we understand the phenomenon of degeneration as circular loops within the directed graph. Based on our formulation, we propose a novel decoding method -- \textit{momentum decoding} -- which encourages the LM to \textit{greedily} explore new nodes outside the current graph. Meanwhile, it also allows the LM to return to the existing nodes with a momentum downgraded by a pre-defined resistance function. We extensively test our approach on three benchmarks from different domains through automatic and human evaluations. The results show that momentum decoding performs comparably with the current state of the art while enjoying notably improved inference speed and computation FLOPs. Furthermore, we conduct a detailed analysis to reveal the merits and inner workings of our approach. Our codes and other related resources are publicly available at https://github.com/gmftbyGMFTBY/MomentumDecoding.