TimeLMs: Diachronic Language Models from Twitter
This addresses the need for time-aware language models for social media analysis, though it is incremental as it builds on existing continual learning approaches.
The authors tackled the problem of time being neglected in NLP by developing TimeLMs, diachronic language models for Twitter data, showing that continual learning improves their ability to handle future and out-of-distribution tweets while making them competitive with benchmarks.
Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models' capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift.