Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
This work addresses a foundational problem in AI by theoretically explaining CoT's effectiveness, offering insights for researchers and practitioners in improving LLM reasoning.
The paper tackles the mystery of why Chain-of-Thought (CoT) prompting improves large language models (LLMs) by providing theoretical proofs that bounded-depth Transformers fail at basic arithmetic tasks without CoT, but constant-size models can solve them using CoT derivations, with experiments showing consistent success in step-by-step solutions.
Recent studies have discovered that Chain-of-Thought prompting (CoT) can dramatically improve the performance of Large Language Models (LLMs), particularly when dealing with complex tasks involving mathematics or reasoning. Despite the enormous empirical success, the underlying mechanisms behind CoT and how it unlocks the potential of LLMs remain elusive. In this paper, we take a first step towards theoretically answering these questions. Specifically, we examine the expressivity of LLMs with CoT in solving fundamental mathematical and decision-making problems. By using circuit complexity theory, we first give impossibility results showing that bounded-depth Transformers are unable to directly produce correct answers for basic arithmetic/equation tasks unless the model size grows super-polynomially with respect to the input length. In contrast, we then prove by construction that autoregressive Transformers of constant size suffice to solve both tasks by generating CoT derivations using a commonly used math language format. Moreover, we show LLMs with CoT can handle a general class of decision-making problems known as Dynamic Programming, thus justifying its power in tackling complex real-world tasks. Finally, an extensive set of experiments show that, while Transformers always fail to directly predict the answers, they can consistently learn to generate correct solutions step-by-step given sufficient CoT demonstrations.