On the Evaluation of Answer-Agnostic Paragraph-level Multi-Question Generation
This work addresses evaluation challenges in question generation for NLP applications, but it is incremental as it builds on existing methods.
The paper tackles the problem of generating multiple questions from a paragraph without answer knowledge, proposing a new evaluation method using the Hungarian algorithm for better coverage assessment and comparing strategies for question generation with pre-trained models.
We study the task of predicting a set of salient questions from a given paragraph without any prior knowledge of the precise answer. We make two main contributions. First, we propose a new method to evaluate a set of predicted questions against the set of references by using the Hungarian algorithm to assign predicted questions to references before scoring the assigned pairs. We show that our proposed evaluation strategy has better theoretical and practical properties compared to prior methods because it can properly account for the coverage of references. Second, we compare different strategies to utilize a pre-trained seq2seq model to generate and select a set of questions related to a given paragraph. The code is available.