QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
This dataset addresses the problem of evaluating NLP systems' ability to comprehend and apply textual qualitative knowledge in novel settings for the NLP community, and it is incremental as it builds on previous datasets by introducing open-domain textual knowledge.
The authors introduced QuaRTz, the first open-domain dataset for reasoning about textual qualitative relationships, and found that state-of-the-art results are 20% below human performance, presenting an open challenge.
We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., "A sunscreen with a higher SPF protects the skin longer.", twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?", plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system's ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20%) below human performance, presenting an open challenge to the NLP community.