Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
This work addresses the challenge of efficiently transferring reasoning abilities from large to small models for cost reduction in tasks like Chain-of-Thought generation, representing an incremental improvement in distillation methods.
The paper tackles the problem of knowledge distillation for reasoning tasks, where students can be overwhelmed by mimicking teacher distributions, and proposes Token-Selective Dual Knowledge Distillation (TSD-KD) to focus on important tokens and encourage self-explanation, achieving state-of-the-art performance with accuracy gains up to 54.4% over baselines and even outperforming the teacher model by up to 20.3% in some cases.
Knowledge Distillation (KD) can transfer the reasoning abilities of large models to smaller ones, which can reduce the costs to generate Chain-of-Thoughts for reasoning tasks. KD methods typically ask the student to mimic the teacher's distribution over the entire output. However, a student with limited capacity can be overwhelmed by such extensive supervision causing a distribution mismatch, especially in complex reasoning tasks. We propose Token-Selective Dual Knowledge Distillation (TSD-KD), a framework for student-centric distillation. TSD-KD focuses on distilling important tokens for reasoning and encourages the student to explain reasoning in its own words. TSD-KD combines indirect and direct distillation. Indirect distillation uses a weak form of feedback based on preference ranking. The student proposes candidate responses generated on its own; the teacher re-ranks those candidates as indirect feedback without enforcing its entire distribution. Direct distillation uses distribution matching; however, it selectively distills tokens based on the relative confidence between teacher and student. Finally, we add entropy regularization to maintain the student's confidence during distillation. Overall, our method provides the student with targeted and indirect feedback to support its own reasoning process and to facilitate self-improvement. The experiments show the state-of-the-art performance of TSD-KD on 10 challenging reasoning benchmarks, outperforming the baseline and runner-up in accuracy by up to 54.4\% and 40.3\%, respectively. Notably, a student trained by TSD-KD even outperformed its own teacher model in four cases by up to 20.3\%. The source code is available at https://github.com/kmswin1/TSD-KD.