Do LLMs Really Think Step-by-step In Implicit Reasoning?
This addresses a key uncertainty in efficient reasoning methods for AI practitioners, revealing limitations in implicit CoT that could impact its adoption.
The study investigated whether implicit Chain-of-Thought reasoning in LLMs truly involves step-by-step thinking, finding that when prompted, LLMs hardly consider intermediate steps and rely on experience, but when trained, they do calculate them, with performance varying by problem format.
It has been well-known that Chain-of-Thought can remarkably enhance LLMs' performance on complex tasks. However, because it also introduces slower inference speeds and higher computational costs, many researches have attempted to use implicit CoT, which does not need LLMs to explicitly generate the intermediate steps. However, the invisible reasoning process leaves us a doubt that, can implicit CoT really be equal to explicit CoT? Therefore, in this study, we address this question through experiments. We probe the information of intermediate steps from the model's hidden states when it is either trained or prompted to perform implicit CoT. The results surprisingly indicate that when prompted, LLMs hardly think about intermediate steps, suggesting they may just rely on experience rather than strict step-by-step reasoning. But when trained, they indeed calculate intermediate steps. Moreover, in both situations, we find the effect of using implicit CoT is susceptible to the format of the problem, reaffirming the current deficiency of implicit CoT.