Prompt, Condition, and Generate: Classification of Unsupported Claims with In-Context Learning
This work addresses the challenge of making sense of unsupported claims for applications like fact-checking, though it is incremental in applying existing in-context learning methods to a new dataset.
The paper tackles the problem of classifying unsupported claims by introducing a new task of distilling narratives from fine-grained debate topics, using a crowdsourced dataset of 12 topics with over 120k annotated arguments. It finds that using large language models to generate claims with evidence improves narrative classification performance and enables inference of stance and aspect with few examples.
Unsupported and unfalsifiable claims we encounter in our daily lives can influence our view of the world. Characterizing, summarizing, and -- more generally -- making sense of such claims, however, can be challenging. In this work, we focus on fine-grained debate topics and formulate a new task of distilling, from such claims, a countable set of narratives. We present a crowdsourced dataset of 12 controversial topics, comprising more than 120k arguments, claims, and comments from heterogeneous sources, each annotated with a narrative label. We further investigate how large language models (LLMs) can be used to synthesise claims using In-Context Learning. We find that generated claims with supported evidence can be used to improve the performance of narrative classification models and, additionally, that the same model can infer the stance and aspect using a few training examples. Such a model can be useful in applications which rely on narratives , e.g. fact-checking.