MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants
This addresses the deployment challenge of large language models by enhancing reasoning in smaller models, though it is incremental as it builds on existing distillation methods.
The paper tackles the problem of small language models (SLMs) struggling to learn long-form chain-of-thought (CoT) reasoning due to limited capacity, known as the 'SLMs Learnability Gap', by introducing MiCoTA, a framework using intermediate-sized models and CoT sequences to bridge gaps, resulting in improvements such as Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct achieving average score increases of 3.47 and 3.93 on benchmarks like AIME2024 and GSM8K.
Large language models (LLMs) excel at reasoning tasks requiring long thought sequences for planning, reflection, and refinement. However, their substantial model size and high computational demands are impractical for widespread deployment. Yet, small language models (SLMs) often struggle to learn long-form CoT reasoning due to their limited capacity, a phenomenon we refer to as the "SLMs Learnability Gap". To address this, we introduce \textbf{Mi}d-\textbf{Co}T \textbf{T}eacher \textbf{A}ssistant Distillation (MiCoTAl), a framework for improving long CoT distillation for SLMs. MiCoTA employs intermediate-sized models as teacher assistants and utilizes intermediate-length CoT sequences to bridge both the capacity and reasoning length gaps. Our experiments on downstream tasks demonstrate that although SLMs distilled from large teachers can perform poorly, by applying MiCoTA, they achieve significant improvements in reasoning performance. Specifically, Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct achieve an improvement of 3.47 and 3.93 respectively on average score on AIME2024, AMC, Olympiad, MATH-500 and GSM8K benchmarks. To better understand the mechanism behind MiCoTA, we perform a quantitative experiment demonstrating that our method produces data more closely aligned with base SLM distributions. Our insights pave the way for future research into long-CoT data distillation for SLMs.