Multi-task Pairwise Neural Ranking for Hashtag Segmentation
This work addresses the challenge of interpreting hashtag semantics for social media applications, offering incremental improvements over existing methods.
The paper tackled the problem of hashtag segmentation by framing it as a pairwise ranking problem and building a dataset of 12,594 hashtags, achieving a 24.6% error reduction in accuracy compared to the state-of-the-art method. It also showed that segmentation improves downstream tasks like sentiment analysis, with a 2.6% increase in average recall on the SemEval 2017 dataset.
Hashtags are often employed on social media and beyond to add metadata to a textual utterance with the goal of increasing discoverability, aiding search, or providing additional semantics. However, the semantic content of hashtags is not straightforward to infer as these represent ad-hoc conventions which frequently include multiple words joined together and can include abbreviations and unorthodox spellings. We build a dataset of 12,594 hashtags split into individual segments and propose a set of approaches for hashtag segmentation by framing it as a pairwise ranking problem between candidate segmentations. Our novel neural approaches demonstrate 24.6% error reduction in hashtag segmentation accuracy compared to the current state-of-the-art method. Finally, we demonstrate that a deeper understanding of hashtag semantics obtained through segmentation is useful for downstream applications such as sentiment analysis, for which we achieved a 2.6% increase in average recall on the SemEval 2017 sentiment analysis dataset.