From task structures to world models: What do LLMs know?
This work addresses a foundational problem in AI and cognitive science regarding the nature of knowledge in LLMs, which is incremental as it builds on existing debates without introducing new methods or data.
The paper investigates the nature of knowledge in large language models (LLMs), proposing 'instrumental knowledge' based on abilities and exploring its relation to human-like 'worldly knowledge' and structured world models. It suggests that LLMs might recover worldly knowledge through a resource-rational tradeoff between world models and task demands.
In what sense does a large language model have knowledge? The answer to this question extends beyond the capabilities of a particular AI system, and challenges our assumptions about the nature of knowledge and intelligence. We answer by granting LLMs "instrumental knowledge"; knowledge defined by a certain set of abilities. We then ask how such knowledge is related to the more ordinary, "worldly" knowledge exhibited by human agents, and explore this in terms of the degree to which instrumental knowledge can be said to incorporate the structured world models of cognitive science. We discuss ways LLMs could recover degrees of worldly knowledge, and suggest such recovery will be governed by an implicit, resource-rational tradeoff between world models and task demands.