Language Models as Measurement Apparatus for Culture
For NLP researchers and cultural analysts, this paper reframes the use of language models in cultural measurement as a material-discursive practice, highlighting the ethical and methodological implications of agential cuts.
This paper argues that language models used to measure cultural phenomena are not passive instruments but actively constitute the cultural reality they measure, drawing on Karen Barad's concept of the agential cut. Through case studies on TV and film dialogue, it demonstrates how design choices in models, data, and evaluation shape cultural measurements, proposing a theory-driven, empirically rigorous research program.
Language models are increasingly used to quantify cultural phenomena, but what makes such measurement distinctively cultural? This paper argues that NLP work on culture is a material-discursive practice: the apparatus -- model, data, annotation, evaluation -- participates in constituting the cultural reality it measures, rather than passively recording it. Drawing on Karen Barad's concept of the agential cut -- the contingent boundary between phenomenon and instrument -- I show that the apparatus's substantive design choices draw such boundaries, and that the boundary is entangled from the start because language models have already internalized much of the cultural material they measure. I illustrate this through three case studies on television and film dialogue (measuring structure, interaction, and deviation) and three examinations of the apparatus itself (erasure of cultural markers, attunement to historical material, and agency in an agentic workflow). This big picture analysis proposes a research program that is theory-driven, empirically rigorous, and culturally contingent, treating each agential cut as a conscious commitment, at once methodological and ethical.