Using cognitive models to reveal value trade-offs in language models
This work addresses the challenge of understanding and controlling value trade-offs in LLMs for developers and researchers, offering a flexible probing tool, though it is incremental as it applies existing cognitive models to new model settings.
The study tackled the problem of interpreting value trade-offs in language models by applying a cognitive model of polite speech to evaluate utility functions in both frontier black-box models and open-source RL-trained models, revealing patterns such as higher informational utility than social utility in default behavior and predictable shifts with goal prioritization, with findings showing large early training shifts in utility values influenced by base model and pretraining data.
Value trade-offs are an integral part of human decision-making and language use, however, current tools for interpreting such dynamic and multi-faceted notions of values in LLMs are limited. In cognitive science, so-called "cognitive models" provide formal accounts of such trade-offs in humans, by modeling the weighting of a speaker's competing utility functions in choosing an action or utterance. Here we use a leading cognitive model of polite speech to systematically evaluate value trade-offs in two encompassing model settings: degrees of reasoning "effort" in frontier black-box models, and RL post-training dynamics of open-source models. Our results highlight patterns of higher informational utility than social utility in reasoning models' default behavior, and demonstrate that these patterns shift in predictable ways when models are prompted to prioritize certain goals over others. Our findings from LLMs' training dynamics suggest large shifts in utility values early on in training with persistent effects of the choice of base model and pretraining data, compared to feedback dataset or alignment method. Our framework offers a flexible tool for probing value trade-offs across diverse model types, providing insights for generating hypotheses about other social behaviors such as sycophancy and for shaping training regimes that better control trade-offs between values during model development.