Chain of Execution Supervision Promotes General Reasoning in Large Language Models
This work addresses the challenge of enhancing reasoning capabilities in LLMs for developers and researchers, though it is incremental as it builds on existing methods like chain-of-thought and code-based training.
The paper tackles the problem of improving general reasoning in large language models by addressing the suboptimal training on raw code due to implicit reasoning and noise, introducing TracePile, a corpus of 2.6 million samples that transforms code into explicit step-by-step rationales, which boosts models like LLaMA3.1-8B by 7.1% on average across nine math datasets.
Building robust and general reasoning ability is a central goal in the development of large language models (LLMs). Recent efforts increasingly turn to code as a rich training source, given its inherent logical structure and diverse reasoning paradigms such as divide-and-conquer, topological ordering, and enumeration. However, reasoning in code is often expressed implicitly and entangled with syntactic or implementation noise, making direct training on raw code suboptimal.To address this, we introduce TracePile, a large-scale corpus of 2.6 million samples that transforms code execution into explicit, step-by-step chain-of-thought-style rationales, which we call Chain of Execution (CoE). The corpus spans domains including mathematics, classical algorithms and algorithmic competition, and is enriched with variable-tracing questions and code rewritings to enhance logical granularity and code diversity. We evaluate TracePile using three training setups: continue-pretraining, instruction tuning after pretraining, and two-stage finetuning. Experiments across four base models (LLaMA 3, LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) and 20 benchmarks covering math, code, logic, and algorithms demonstrate consistent improvements. Notably, TracePile boosts LLaMA3.1-8B by 7.1\% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and MMLU under two-stage fine-tuning.