A Comparative Study on Collecting High-Quality Implicit Reasonings at a Large-scale
This addresses the problem of low-quality warrants for natural language understanding systems, though it appears incremental as it focuses on improving collection methods rather than a breakthrough.
The paper tackles the challenge of collecting high-quality implicit reasonings (warrants) in arguments by devising various methodologies, resulting in a preliminary dataset of 6,000 warrants annotated over 600 arguments for 3 debatable topics.
Explicating implicit reasoning (i.e. warrants) in arguments is a long-standing challenge for natural language understanding systems. While recent approaches have focused on explicating warrants via crowdsourcing or expert annotations, the quality of warrants has been questionable due to the extreme complexity and subjectivity of the task. In this paper, we tackle the complex task of warrant explication and devise various methodologies for collecting warrants. We conduct an extensive study with trained experts to evaluate the resulting warrants of each methodology and find that our methodologies allow for high-quality warrants to be collected. We construct a preliminary dataset of 6,000 warrants annotated over 600 arguments for 3 debatable topics. To facilitate research in related downstream tasks, we release our guidelines and preliminary dataset.