Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
This addresses a conceptual problem for AI researchers and practitioners by highlighting the risks of misleading metaphors in model interpretation.
The paper argues that anthropomorphizing intermediate tokens in language models as 'reasoning traces' is harmful, as it misrepresents model behavior and leads to questionable research practices.
Intermediate token generation (ITG), where a model produces output before the solution, has been proposed as a method to improve the performance of language models on reasoning tasks. These intermediate tokens have been called "reasoning traces" or even "thoughts" -- implicitly anthropomorphizing the model, implying these tokens resemble steps a human might take when solving a challenging problem.In this paper, we present evidence that this anthropomorphization isn't a harmless metaphor, and instead is quite dangerous -- it confuses the nature of these models and how to use them effectively, and leads to questionable research.