Fine-grained Czech News Article Dataset: An Interdisciplinary Approach to Trustworthiness Analysis
This work addresses the interdisciplinary challenge of disinformation and media literacy by providing a dataset and methodology for trustworthiness analysis, though it is incremental as it applies existing methods to new data.
The authors tackled the problem of assessing news article trustworthiness by creating a novel dataset of over 10,000 Czech news articles with fine-grained annotations across four credibility classes, and they achieved a best testing F-1 score of 0.52 using fine-tuned language models.
We present the Verifee Dataset: a novel dataset of news articles with fine-grained trustworthiness annotations. We develop a detailed methodology that assesses the texts based on their parameters encompassing editorial transparency, journalist conventions, and objective reporting while penalizing manipulative techniques. We bring aboard a diverse set of researchers from social, media, and computer sciences to overcome barriers and limited framing of this interdisciplinary problem. We collect over $10,000$ unique articles from almost $60$ Czech online news sources. These are categorized into one of the $4$ classes across the credibility spectrum we propose, raging from entirely trustworthy articles all the way to the manipulative ones. We produce detailed statistics and study trends emerging throughout the set. Lastly, we fine-tune multiple popular sequence-to-sequence language models using our dataset on the trustworthiness classification task and report the best testing F-1 score of $0.52$. We open-source the dataset, annotation methodology, and annotators' instructions in full length at https://verifee.ai/research to enable easy build-up work. We believe similar methods can help prevent disinformation and educate in the realm of media literacy.