Code Execution as Grounded Supervision for LLM Reasoning
This addresses the problem of scalable and accurate reasoning supervision for LLM training, offering a more efficient alternative to human annotations or error-prone LLM-generated data.
The paper tackles the challenge of obtaining reliable reasoning supervision for large language models by proposing a method that generates high-quality chain-of-thought datasets using verifiable code execution traces, resulting in improved reasoning abilities across diverse tasks and reduced token length during inference.
Training large language models (LLMs) with chain-of-thought (CoT) supervision has proven effective for enhancing their reasoning abilities. However, obtaining reliable and accurate reasoning supervision remains a significant challenge. We propose a scalable method for generating a high-quality CoT supervision dataset by leveraging the determinism of program execution. Unlike existing reasoning dataset generation methods that rely on costly human annotations or error-prone LLM-generated CoT, our approach extracts verifiable, step-by-step reasoning traces from code execution and transforms them into a natural language CoT reasoning. Experiments on reasoning benchmarks across various domains show that our method effectively equips LLMs with transferable reasoning abilities across diverse tasks. Furthermore, the ablation studies validate that our method produces highly accurate reasoning data and reduces overall token length during inference by reducing meaningless repetition and overthinking.