CS563-QA: A Collection for Evaluating Question Answering Systems
This provides a new benchmark for evaluating question answering systems, though it is incremental as it builds on existing evaluation needs.
The authors tackled the need for better evaluation in question answering by creating CS563-QA, a small collection of free text questions with increasing difficulty levels, which can be used for rapid system evaluation and has educational value.
Question Answering (QA) is a challenging topic since it requires tackling the various difficulties of natural language understanding. Since evaluation is important not only for identifying the strong and weak points of the various techniques for QA, but also for facilitating the inception of new methods and techniques, in this paper we present a collection for evaluating QA methods over free text that we have created. Although it is a small collection, it contains cases of increasing difficulty, therefore it has an educational value and it can be used for rapid evaluation of QA systems.