RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media
This addresses the challenge of verifying health information on social media for researchers and practitioners, though it is incremental as it builds on existing annotation and retrieval methods.
The authors tackled the problem of identifying and retrieving trustworthy evidence for medical claims made on social media by creating RedHOT, a corpus of 22,000 annotated Reddit posts across 24 health conditions, and developed a dense retrieval model that outperformed baselines, with manual evaluation by doctors showing promising but improvable results.
We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly annotated social media posts from Reddit spanning 24 health conditions. Annotations include demarcations of spans corresponding to medical claims, personal experiences, and questions. We collect additional granular annotations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence relevant to a given claim made on social media. We propose a new method to automatically derive (noisy) supervision for this task which we use to train a dense retrieval model; this outperforms baseline models. Manual evaluation of retrieval results performed by medical doctors indicate that while our system performance is promising, there is considerable room for improvement. Collected annotations (and scripts to assemble the dataset), are available at https://github.com/sominw/redhot.