Changes in European Solidarity Before and During COVID-19: Evidence from a Large Crowd- and Expert-Annotated Twitter Dataset
This work addresses the problem of tracking social discourse changes for researchers in computational social science, though it is incremental as it applies existing methods to a new domain.
The study tackled the problem of measuring changes in European solidarity discourses before and during the COVID-19 pandemic by applying supervised machine learning to a new NLP problem setting, achieving a 25-point improvement in macro-F1 score from 58% to nearly 85% with an augmented BERT model. The results showed that solidarity became more salient and contested during the crisis, with anti-solidarity tweets spiking initially and then stabilizing at a higher level.
We introduce the well-established social scientific concept of social solidarity and its contestation, anti-solidarity, as a new problem setting to supervised machine learning in NLP to assess how European solidarity discourses changed before and after the COVID-19 outbreak was declared a global pandemic. To this end, we annotate 2.3k English and German tweets for (anti-)solidarity expressions, utilizing multiple human annotators and two annotation approaches (experts vs.\ crowds). We use these annotations to train a BERT model with multiple data augmentation strategies. Our augmented BERT model that combines both expert and crowd annotations outperforms the baseline BERT classifier trained with expert annotations only by over 25 points, from 58\% macro-F1 to almost 85\%. We use this high-quality model to automatically label over 270k tweets between September 2019 and December 2020. We then assess the automatically labeled data for how statements related to European (anti-)solidarity discourses developed over time and in relation to one another, before and during the COVID-19 crisis. Our results show that solidarity became increasingly salient and contested during the crisis. While the number of solidarity tweets remained on a higher level and dominated the discourse in the scrutinized time frame, anti-solidarity tweets initially spiked, then decreased to (almost) pre-COVID-19 values before rising to a stable higher level until the end of 2020.