Whose Values? Measuring the (Subjective) Expression of Basic Human Values in Social Media
This addresses the challenge of value alignment in sociotechnical systems like social media, where algorithmic curation influences exposure to value-laden content, though it is incremental in improving measurement methods.
The paper tackled the problem of measuring subjective human values in social media posts by introducing a personalized framework based on the Schwartz value system, achieving predictions that people agree with more than they agree with other people.
The value alignment of sociotechnical systems has become a central debate, but progress depends on how human values are perceived in the content these systems surface and how such perceptions can be measured at scale. Social media platforms are a prominent class of sociotechnical systems where algorithmic curation shapes exposure to value-laden content at scale. Large-language models offer new opportunities for measuring expressions of human values (e.g., humility or equality) in social media data, but value expressions can be subjective: different people will annotate the same post with different values. In this paper, we draw on the Schwartz value system as a broadly encompassing and theoretically grounded set of basic human values, and introduce a framework to personalize the measurement of expressions of Schwartz values in social media posts at scale. We collect 32,370 ground truth value expression annotations from N=1,079 people on 5,211 social media posts representative of real users' feeds. Due to the subjectivity of the task, we observe low levels of inter-rater agreement between people, and low agreement between human raters and LLM-based methods. In response, we construct a personalization architecture for classifying value expressions by learning from a small number of highly informative calibration annotations per user. In evaluation, we find that modeling these differences successfully yields value expression predictions that people agree with more than they agree with other people. These results contribute new methods and understanding for the measurement of human values in social media data.