Testing Causal Models of Word Meaning in GPT-3 and -4
This work addresses how LLMs represent lexical concepts, which is important for understanding AI cognition, but it is incremental as it applies an existing theory to new models.
The paper tested whether GPT-3 and GPT-4 encode causal structures of word meanings as hypothesized by HIPE theory, finding no evidence for GPT-3 but evidence for GPT-4.
Large Language Models (LLMs) have driven extraordinary improvements in NLP. However, it is unclear how such models represent lexical concepts-i.e., the meanings of the words they use. This paper evaluates the lexical representations of GPT-3 and GPT-4 through the lens of HIPE theory, a theory of concept representations which focuses on representations of words describing artifacts (such as "mop", "pencil", and "whistle"). The theory posits a causal graph that relates the meanings of such words to the form, use, and history of the objects to which they refer. We test LLMs using the same stimuli originally used by Chaigneau et al. (2004) to evaluate the theory in humans, and consider a variety of prompt designs. Our experiments concern judgements about causal outcomes, object function, and object naming. We find no evidence that GPT-3 encodes the causal structure hypothesized by HIPE, but do find evidence that GPT-4 encodes such structure. The results contribute to a growing body of research characterizing the representational capacity of large language models.