SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
This addresses the gap in evaluating and adapting language models for social media, which is socially, economically, and politically important but distinct from standard language.
The authors tackled the problem of transformer-based language models' limited ability to understand social media language by quantifying its distinctiveness and introducing the SMILE benchmark covering four platforms and eleven tasks. They showed that adapting tokenization and pretraining on mixed data yields a model that outperforms the best similar-sized alternative by 4.2 points on the overall SMILE score.
We study the ability of transformer-based language models (LMs) to understand social media language. Social media (SM) language is distinct from standard written language, yet existing benchmarks fall short of capturing LM performance in this socially, economically, and politically important domain. We quantify the degree to which social media language differs from conventional language and conclude that the difference is significant both in terms of token distribution and rate of linguistic shift. Next, we introduce a new benchmark for Social MedIa Language Evaluation (SMILE) that covers four SM platforms and eleven tasks. Finally, we show that learning a tokenizer and pretraining on a mix of social media and conventional language yields an LM that outperforms the best similar-sized alternative by 4.2 points on the overall SMILE score.