Human Language Modeling
This work addresses the limitation of traditional language models in accounting for human variability, offering a novel framework for social media analysis, though it is incremental in extending existing transformer methods.
The paper tackles the problem of modeling language as generated by humans rather than independently, proposing Human Language Modeling (HuLM) with a hierarchical approach to capture human states across document sequences, and results show that their HaRT model achieves state-of-the-art performance on language modeling and four downstream tasks including stance detection and sentiment classification.
Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem whereby a human-level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HuLM task, pre-trained on approximately 100,000 social media users, and demonstrate its effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels: stance detection, sentiment classification, age estimation, and personality assessment. Results on all tasks meet or surpass the current state-of-the-art.