Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
This addresses the issue of spurious memorization affecting reasoning reliability in LLMs, offering a diagnostic tool for researchers and practitioners, though it is incremental in improving model analysis.
The paper tackles the problem of memorization in Chain-of-Thought reasoning by introducing STIM, a framework that identifies token-level memorization sources, revealing that up to 67% of wrong tokens are driven by local memorization in complex cases.
Large Language Models (LLMs) perform well on reasoning benchmarks but often fail when inputs alter slightly, raising concerns about the extent to which their success relies on memorization. This issue is especially acute in Chain-of-Thought (CoT) reasoning, where spurious memorized patterns can trigger intermediate errors that cascade into incorrect final answers. We introduce STIM, a novel framework for Source-aware Token-level Identification of Memorization, which attributes each token in a reasoning chain to one of multiple memorization sources - local, mid-range, or long-range - based on their statistical co-occurrence with the token in the pretraining corpus. Our token-level analysis across tasks and distributional settings reveals that models rely more on memorization in complex or long-tail cases, and that local memorization is often the dominant driver of errors, leading to up to 67% of wrong tokens. We also show that memorization scores from STIM can be effective in predicting the wrong tokens in the wrong reasoning step. STIM offers a powerful tool for diagnosing and improving model reasoning and can generalize to other structured step-wise generation tasks.