When Researchers Say Mental Model/Theory of Mind of AI, What Are They Really Talking About?
It critiques the current discourse on AI cognition as incremental, highlighting flaws in testing paradigms for researchers in AI and cognitive science.
This position paper argues that claims about AI possessing Theory of Mind or mental models are based on behavioral predictions and bias corrections rather than genuine cognition, and suggests shifting focus to mutual frameworks that account for human-AI interaction dynamics.
When researchers claim AI systems possess ToM or mental models, they are fundamentally discussing behavioral predictions and bias corrections rather than genuine mental states. This position paper argues that the current discourse conflates sophisticated pattern matching with authentic cognition, missing a crucial distinction between simulation and experience. While recent studies show LLMs achieving human-level performance on ToM laboratory tasks, these results are based only on behavioral mimicry. More importantly, the entire testing paradigm may be flawed in applying individual human cognitive tests to AI systems, but assessing human cognition directly in the moment of human-AI interaction. I suggest shifting focus toward mutual ToM frameworks that acknowledge the simultaneous contributions of human cognition and AI algorithms, emphasizing the interaction dynamics, instead of testing AI in isolation.