SkillQG: Learning to Generate Question for Reading Comprehension Assessment
This work addresses the need for fine-grained assessment and improvement of machine reading comprehension models by generating questions with controllable comprehension types, representing an incremental advance over existing methods that focus on literal information.
The authors tackled the problem of generating questions for reading comprehension assessment by introducing SkillQG, a framework that controls comprehension types based on a hierarchical skill-based schema, resulting in outperforming baselines in quality, relevance, and skill-controllability, and showing a performance boost in downstream question answering tasks.
We present $\textbf{$\texttt{SkillQG}$}$: a question generation framework with controllable comprehension types for assessing and improving machine reading comprehension models. Existing question generation systems widely differentiate questions by $\textit{literal}$ information such as question words and answer types to generate semantically relevant questions for a given context. However, they rarely consider the $\textit{comprehension}$ nature of questions, i.e. the different comprehension capabilities embodied by different questions. In comparison, our $\texttt{SkillQG}$ is able to tailor a fine-grained assessment and improvement to the capabilities of question answering models built on it. Specifically, we first frame the comprehension type of questions based on a hierarchical skill-based schema, then formulate $\texttt{SkillQG}$ as a skill-conditioned question generator. Furthermore, to improve the controllability of generation, we augment the input text with question focus and skill-specific knowledge, which are constructed by iteratively prompting the pre-trained language models. Empirical results demonstrate that $\texttt{SkillQG}$ outperforms baselines in terms of quality, relevance, and skill-controllability while showing a promising performance boost in downstream question answering task.