Hierarchical Aligned Multimodal Learning for NER on Tweet Posts
This work addresses the problem of extracting structured knowledge from multimodal tweets for applications like recommendation, though it appears incremental in advancing MNER methods.
The paper tackles multimodal named entity recognition (MNER) on tweet posts by proposing a hierarchical aligned learning approach that dynamically aligns image and text sequences for cross-modal learning, achieving improved performance as demonstrated on two open datasets.
Mining structured knowledge from tweets using named entity recognition (NER) can be beneficial for many down stream applications such as recommendation and intention understanding. With tweet posts tending to be multimodal, multimodal named entity recognition (MNER) has attracted more attention. In this paper, we propose a novel approach, which can dynamically align the image and text sequence and achieve the multi-level cross-modal learning to augment textual word representation for MNER improvement. To be specific, our framework can be split into three main stages: the first stage focuses on intra-modality representation learning to derive the implicit global and local knowledge of each modality, the second evaluates the relevance between the text and its accompanying image and integrates different grained visual information based on the relevance, the third enforces semantic refinement via iterative cross-modal interactions and co-attention. We conduct experiments on two open datasets, and the results and detailed analysis demonstrate the advantage of our model.