Transformers, Contextualism, and Polysemy
This work addresses philosophical problems in linguistics and AI for researchers interested in meaning and context, but it is incremental as it builds on existing transformer technology without introducing new methods or data.
The paper tackles the problem of understanding the relationship between context and meaning in natural language by proposing a theory derived from the transformer architecture, arguing that it offers novel insights into philosophical debates on contextualism and polysemy.
The transformer architecture, introduced by Vaswani et al. (2017), is at the heart of the remarkable recent progress in the development of language models, including widely-used chatbots such as Chat-GPT and Claude. In this paper, I argue that we can extract from the way the transformer architecture works a theory of the relationship between context and meaning. I call this the transformer theory, and I argue that it is novel with regard to two related philosophical debates: the contextualism debate regarding the extent of context-sensitivity across natural language, and the polysemy debate regarding how polysemy should be captured within an account of word meaning.