Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?
This work addresses the need for more personalized language models in human-centered NLP, though it is incremental as it builds on existing methods without introducing a new paradigm.
The study tackled the problem of incorporating human context into pre-trained language models by comparing group attributes, individual traits, and a combined approach on five tasks, finding no single best method but highlighting potential avenues for human-centered modeling.
Pre-trained language models consider the context of neighboring words and documents but lack any author context of the human generating the text. However, language depends on the author's states, traits, social, situational, and environmental attributes, collectively referred to as human context (Soni et al., 2024). Human-centered natural language processing requires incorporating human context into language models. Currently, two methods exist: pre-training with 1) group-wise attributes (e.g., over-45-year-olds) or 2) individual traits. Group attributes are simple but coarse -- not all 45-year-olds write the same way -- while individual traits allow for more personalized representations, but require more complex modeling and data. It is unclear which approach benefits what tasks. We compare pre-training models with human context via 1) group attributes, 2) individual users, and 3) a combined approach on five user- and document-level tasks. Our results show that there is no best approach, but that human-centered language modeling holds avenues for different methods.