Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
This work addresses the challenge of Chinese machine reading comprehension for language learners and AI systems, but it is incremental as it focuses on dataset creation and analysis rather than a novel method.
The authors tackled the problem of machine reading comprehension in Chinese by creating the C^3 dataset, which includes 13,369 documents and 19,577 multiple-choice questions, and found that the best model achieved 68.5% accuracy compared to 96.0% for humans, with 86.8% of questions requiring prior knowledge beyond the text.
Machine reading comprehension tasks require a machine reader to answer questions relevant to the given document. In this paper, we present the first free-form multiple-Choice Chinese machine reading Comprehension dataset (C^3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese-as-a-second-language examinations. We present a comprehensive analysis of the prior knowledge (i.e., linguistic, domain-specific, and general world knowledge) needed for these real-world problems. We implement rule-based and popular neural methods and find that there is still a significant performance gap between the best performing model (68.5%) and human readers (96.0%), especially on problems that require prior knowledge. We further study the effects of distractor plausibility and data augmentation based on translated relevant datasets for English on model performance. We expect C^3 to present great challenges to existing systems as answering 86.8% of questions requires both knowledge within and beyond the accompanying document, and we hope that C^3 can serve as a platform to study how to leverage various kinds of prior knowledge to better understand a given written or orally oriented text. C^3 is available at https://dataset.org/c3/.