The future is different: Large pre-trained language models fail in prediction tasks
This addresses a critical issue for AI systems relying on LPLMs in dynamic real-world applications like social media analysis, though it is incremental as it builds on existing topic modeling and attention techniques.
The paper tackles the problem of large pre-trained language models (LPLMs) failing under temporal distribution shifts, showing they drop by about 88% in predicting future post popularity on Reddit datasets, and introduces a method that reduces drops to 40% worst-case while using only 7% of parameters.
Large pre-trained language models (LPLM) have shown spectacular success when fine-tuned on downstream supervised tasks. Yet, it is known that their performance can drastically drop when there is a distribution shift between the data used during training and that used at inference time. In this paper we focus on data distributions that naturally change over time and introduce four new REDDIT datasets, namely the WALLSTREETBETS, ASKSCIENCE, THE DONALD, and POLITICS sub-reddits. First, we empirically demonstrate that LPLM can display average performance drops of about 88% (in the best case!) when predicting the popularity of future posts from sub-reddits whose topic distribution changes with time. We then introduce a simple methodology that leverages neural variational dynamic topic models and attention mechanisms to infer temporal language model representations for regression tasks. Our models display performance drops of only about 40% in the worst cases (2% in the best ones) when predicting the popularity of future posts, while using only about 7% of the total number of parameters of LPLM and providing interpretable representations that offer insight into real-world events, like the GameStop short squeeze of 2021