Limits to Predicting Online Speech Using Large Language Models
This addresses the challenge of modeling user-generated content for applications like personalization or content moderation, but it is incremental as it builds on existing predictability studies with new data and context analysis.
The paper tackles the problem of predicting individual users' online speech on X (formerly Twitter) using large language models, finding that it remains surprisingly hard, with models using users' own history significantly outperforming those using social circle posts, validated across models up to 70 billion parameters.
Our paper studies the predictability of online speech -- that is, how well language models learn to model the distribution of user generated content on X (previously Twitter). We define predictability as a measure of the model's uncertainty, i.e. its negative log-likelihood. As the basis of our study, we collect 10M tweets for ``tweet-tuning'' base models and a further 6.25M posts from more than five thousand X (previously Twitter) users and their peers. In our study involving more than 5000 subjects, we find that predicting posts of individual users remains surprisingly hard. Moreover, it matters greatly what context is used: models using the users' own history significantly outperform models using posts from their social circle. We validate these results across four large language models ranging in size from 1.5 billion to 70 billion parameters. Moreover, our results replicate if instead of prompting the model with additional context, we finetune on it. We follow up with a detailed investigation on what is learned in-context and a demographic analysis. Up to 20\% of what is learned in-context is the use of @-mentions and hashtags. Our main results hold across the demographic groups we studied.