Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model
This work addresses the challenge of computational inefficiency in deploying LLMs for mathematical reasoning, offering a method to enhance SLMs for more efficient AI applications, though it appears incremental by building on prior distillation techniques.
The paper tackles the problem of deploying large language models (LLMs) for mathematical reasoning by distilling their capabilities into smaller language models (SLMs) to reduce computational demands, proposing Key-Point-Driven Mathematical Reasoning Distillation (KPDD) which breaks down problem-solving into stages, with KPDD-CoT improving reasoning abilities and KPDD-PoT achieving state-of-the-art performance in mathematical reasoning tasks.
Large Language Models (LLMs) have demonstrated exceptional proficiency in mathematical reasoning tasks due to their extensive parameter counts and training on vast datasets. Despite these capabilities, deploying LLMs is hindered by their computational demands. Distilling LLM mathematical reasoning into Smaller Language Models (SLMs) has emerged as a solution to this challenge, although these smaller models often suffer from errors in calculation and semantic understanding. Prior work has proposed Program-of-Thought Distillation (PoTD) to avoid calculation error. To further address semantic understanding errors, we propose Key-Point-Driven Mathematical Reasoning Distillation (KPDD). KPDD enhances the reasoning performance of SLMs by breaking down the problem-solving process into three stages: Core Question Extraction, Problem-Solving Information Extraction, and Step-by-Step Solution. This method is further divided into KPDD-CoT, which generates Chain-of-Thought rationales, and KPDD-PoT, which creates Program-of-Thought rationales. The experiment results show that KPDD-CoT significantly improves reasoning abilities, while KPDD-PoT achieves state-of-the-art performance in mathematical reasoning tasks. Our approach effectively mitigates misunderstanding errors, advancing the deployment of efficient and capable SLMs.