Backward and Forward Language Modeling for Constrained Sentence Generation
This addresses a limitation in language generation for applications like machine translation and conversation systems where specific words must be included, though it is an incremental improvement over existing methods.
The paper tackles the problem of generating sentences with hard constraints, such as requiring a specific word to appear, by proposing a backward and forward language model using RNNs to generate words around the given word. Experimental results show that the generated texts achieve quality comparable to sequential language models.
Recent language models, especially those based on recurrent neural networks (RNNs), make it possible to generate natural language from a learned probability. Language generation has wide applications including machine translation, summarization, question answering, conversation systems, etc. Existing methods typically learn a joint probability of words conditioned on additional information, which is (either statically or dynamically) fed to RNN's hidden layer. In many applications, we are likely to impose hard constraints on the generated texts, i.e., a particular word must appear in the sentence. Unfortunately, existing approaches could not solve this problem. In this paper, we propose a novel backward and forward language model. Provided a specific word, we use RNNs to generate previous words and future words, either simultaneously or asynchronously, resulting in two model variants. In this way, the given word could appear at any position in the sentence. Experimental results show that the generated texts are comparable to sequential LMs in quality.