Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling
This work addresses the problem of inconsistent sentiment annotations for researchers and practitioners in sentiment analysis and related fields, representing an incremental improvement in annotation methodology.
The study tackled the challenge of obtaining consistent fine-grained sentiment association scores for words by applying Best-Worst Scaling across three domains, showing high consistency in word rankings across different annotators and determining the minimum perceptible difference in sentiment for native speakers.
Access to word-sentiment associations is useful for many applications, including sentiment analysis, stance detection, and linguistic analysis. However, manually assigning fine-grained sentiment association scores to words has many challenges with respect to keeping annotations consistent. We apply the annotation technique of Best-Worst Scaling to obtain real-valued sentiment association scores for words and phrases in three different domains: general English, English Twitter, and Arabic Twitter. We show that on all three domains the ranking of words by sentiment remains remarkably consistent even when the annotation process is repeated with a different set of annotators. We also, for the first time, determine the minimum difference in sentiment association that is perceptible to native speakers of a language.