Modeling Subjectivity in Cognitive Appraisal with Language Models
This work addresses the challenge of modeling subjective preferences in NLP for cognitive science applications, representing an incremental step at the intersection of these fields.
The paper tackled the problem of quantifying subjectivity in cognitive appraisal using language models, finding that personality traits and demographic information are critical for measuring subjectivity, but existing post-hoc calibration methods often fail to achieve satisfactory performance.
As the utilization of language models in interdisciplinary, human-centered studies grow, expectations of their capabilities continue to evolve. Beyond excelling at conventional tasks, models are now expected to perform well on user-centric measurements involving confidence and human (dis)agreement-factors that reflect subjective preferences. While modeling subjectivity plays an essential role in cognitive science and has been extensively studied, its investigation at the intersection with NLP remains under-explored. In light of this gap, we explore how language models can quantify subjectivity in cognitive appraisal by conducting comprehensive experiments and analyses with both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative results demonstrate that personality traits and demographic information are critical for measuring subjectivity, yet existing post-hoc calibration methods often fail to achieve satisfactory performance. Furthermore, our in-depth analysis provides valuable insights to guide future research at the intersection of NLP and cognitive science.