GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
This provides a new benchmark for evaluating NLP models on scenario-based question answering in geography, addressing a domain-specific need for high school education and research.
The authors introduced GeoSQA, a dataset of 1,981 scenarios and 4,110 multiple-choice questions for scenario-based question answering in high school geography, with annotated diagrams to aid NLP research, and benchmarked it against state-of-the-art methods to highlight its challenges.
Scenario-based question answering (SQA) has attracted increasing research attention. It typically requires retrieving and integrating knowledge from multiple sources, and applying general knowledge to a specific case described by a scenario. SQA widely exists in the medical, geography, and legal domains---both in practice and in the exams. In this paper, we introduce the GeoSQA dataset. It consists of 1,981 scenarios and 4,110 multiple-choice questions in the geography domain at high school level, where diagrams (e.g., maps, charts) have been manually annotated with natural language descriptions to benefit NLP research. Benchmark results on a variety of state-of-the-art methods for question answering, textual entailment, and reading comprehension demonstrate the unique challenges presented by SQA for future research.