Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis
This work addresses the problem of noisy and colloquial social media data for NLP researchers and practitioners by providing an annotated dataset and pre-trained models, though it is incremental as it builds on existing frameworks and datasets.
The authors tackled the lack of a comprehensive training corpus for both Named Entity Recognition (NER) and syntactic analysis in tweets by creating Tweebank-NER, an annotated English NER dataset based on Tweebank V2, and developed NLP models including Twitter-Stanza and BERTweet-based models, achieving state-of-the-art performance in tokenization and lemmatization, with competitive results in NER, POS tagging, and dependency parsing against non-transformer models, and new SOTA in POS tagging and dependency parsing using transformer models.
Social media data such as Twitter messages ("tweets") pose a particular challenge to NLP systems because of their short, noisy, and colloquial nature. Tasks such as Named Entity Recognition (NER) and syntactic parsing require highly domain-matched training data for good performance. To date, there is no complete training corpus for both NER and syntactic analysis (e.g., part of speech tagging, dependency parsing) of tweets. While there are some publicly available annotated NLP datasets of tweets, they are only designed for individual tasks. In this study, we aim to create Tweebank-NER, an English NER corpus based on Tweebank V2 (TB2), train state-of-the-art (SOTA) Tweet NLP models on TB2, and release an NLP pipeline called Twitter-Stanza. We annotate named entities in TB2 using Amazon Mechanical Turk and measure the quality of our annotations. We train the Stanza pipeline on TB2 and compare with alternative NLP frameworks (e.g., FLAIR, spaCy) and transformer-based models. The Stanza tokenizer and lemmatizer achieve SOTA performance on TB2, while the Stanza NER tagger, part-of-speech (POS) tagger, and dependency parser achieve competitive performance against non-transformer models. The transformer-based models establish a strong baseline in Tweebank-NER and achieve the new SOTA performance in POS tagging and dependency parsing on TB2. We release the dataset and make both the Stanza pipeline and BERTweet-based models available "off-the-shelf" for use in future Tweet NLP research. Our source code, data, and pre-trained models are available at: \url{https://github.com/social-machines/TweebankNLP}.