Understanding Chain-of-Thought in LLMs through Information Theory
This provides a more accurate evaluation method for CoT reasoning in LLMs, addressing limitations in current techniques for researchers and practitioners.
The paper tackles the problem of evaluating Chain-of-Thought reasoning in LLMs by developing an information-theoretic framework that quantifies information gain at each step, enabling failure mode identification without annotated data. It demonstrates efficacy on datasets like GSM8K and PRM800k, outperforming existing outcome-based methods.
Large Language Models (LLMs) have shown impressive performance in complex reasoning tasks through the use of Chain-of-Thought (CoT) reasoning, allowing models to break down problems into manageable sub-tasks. However, existing CoT evaluation techniques either require annotated CoT data or fall short in accurately assessing intermediate reasoning steps, leading to high rates of false positives. In this paper, we formalize CoT reasoning in LLMs through an information-theoretic lens. Specifically, our framework quantifies the `information-gain' at each reasoning step, enabling the identification of failure modes in LLMs without the need for expensive annotated datasets. We demonstrate the efficacy of our approach through extensive experiments on toy arithmetic, GSM8K and PRM800k datasets, where it significantly outperforms existing outcome-based methods by providing more accurate insights into model performance on individual subtasks.