Evaluation of Question Generation Needs More References
This work addresses a specific evaluation bottleneck in natural language processing for question generation, offering an incremental improvement in assessment accuracy.
The paper tackles the problem of evaluating question generation methods, which typically rely on single-reference metrics, by proposing the use of paraphrased references to create multiple pseudo-references. The result shows that this approach improves evaluation robustness, with higher correlation to human judgments compared to single-reference methods.
Question generation (QG) is the task of generating a valid and fluent question based on a given context and the target answer. According to various purposes, even given the same context, instructors can ask questions about different concepts, and even the same concept can be written in different ways. However, the evaluation for QG usually depends on single reference-based similarity metrics, such as n-gram-based metric or learned metric, which is not sufficient to fully evaluate the potential of QG methods. To this end, we propose to paraphrase the reference question for a more robust QG evaluation. Using large language models such as GPT-3, we created semantically and syntactically diverse questions, then adopt the simple aggregation of the popular evaluation metrics as the final scores. Through our experiments, we found that using multiple (pseudo) references is more effective for QG evaluation while showing a higher correlation with human evaluations than evaluation with a single reference.