Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation
This work addresses a specific limitation in question generation for NLP applications, representing an incremental improvement over existing pre-trained models.
The paper tackled the problem of pre-trained language models ignoring text structure in question generation by modeling answer position and syntactic dependencies, achieving competitive results with state-of-the-art models on the SQuAD dataset.
Today the pre-trained language models achieve great success for question generation (QG) task and significantly outperform traditional sequence-to-sequence approaches. However, the pre-trained models treat the input passage as a flat sequence and are thus not aware of the text structure of input passage. For QG task, we model text structure as answer position and syntactic dependency, and propose answer localness modeling and syntactic mask attention to address these limitations. Specially, we present localness modeling with a Gaussian bias to enable the model to focus on answer-surrounded context, and propose a mask attention mechanism to make the syntactic structure of input passage accessible in question generation process. Experiments on SQuAD dataset show that our proposed two modules improve performance over the strong pre-trained model ProphetNet, and combing them together achieves very competitive results with the state-of-the-art pre-trained model.