A multimodal deep learning approach for named entity recognition from social media
This work addresses the problem of noisy content in social media for NLP researchers and practitioners, offering incremental improvements through multimodal integration.
The paper tackles named entity recognition from noisy social media posts by proposing two multimodal deep learning approaches that combine image and text features, resulting in improved precision, recall, and F1 scores compared to state-of-the-art methods.
Named Entity Recognition (NER) from social media posts is a challenging task. User generated content that forms the nature of social media, is noisy and contains grammatical and linguistic errors. This noisy content makes it much harder for tasks such as named entity recognition. We propose two novel deep learning approaches utilizing multimodal deep learning and Transformers. Both of our approaches use image features from short social media posts to provide better results on the NER task. On the first approach, we extract image features using InceptionV3 and use fusion to combine textual and image features. This presents more reliable name entity recognition when the images related to the entities are provided by the user. On the second approach, we use image features combined with text and feed it into a BERT like Transformer. The experimental results, namely, the precision, recall and F1 score metrics show the superiority of our work compared to other state-of-the-art NER solutions.