Searching for Structure in Unfalsifiable Claims
This work addresses the challenge of analyzing unfalsifiable claims on social media for researchers and platform moderators, but it is incremental as it builds on existing topic modeling and human-in-the-loop approaches.
The paper tackles the problem of distilling unfalsifiable claims from social media into a small set of narratives to facilitate informed debates, introducing a dataset and a human-in-the-loop pipeline that outperforms recent large transformer models and state-of-the-art unsupervised topic models.
Social media platforms give rise to an abundance of posts and comments on every topic imaginable. Many of these posts express opinions on various aspects of society, but their unfalsifiable nature makes them ill-suited to fact-checking pipelines. In this work, we aim to distill such posts into a small set of narratives that capture the essential claims related to a given topic. Understanding and visualizing these narratives can facilitate more informed debates on social media. As a first step towards systematically identifying the underlying narratives on social media, we introduce PAPYER, a fine-grained dataset of online comments related to hygiene in public restrooms, which contains a multitude of unfalsifiable claims. We present a human-in-the-loop pipeline that uses a combination of machine and human kernels to discover the prevailing narratives and show that this pipeline outperforms recent large transformer models and state-of-the-art unsupervised topic models.