Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility
For practitioners of reasoning distillation, this work provides a principled way to select training data that adapts to the student model, leading to better performance.
This paper introduces the Data-Model Compatibility (DMC) metric to assess dataset suitability for reasoning distillation, showing strong correlation with performance and that dynamic DMC-based data selection improves results across multiple models and tasks.
Reasoning distillation transfers complex reasoning abilities from large language models (LLMs) to smaller ones, yet its success depends on how well the training data align with the student model. This paper introduces the Data-Model Compatibility (DMC) metric, which can be used to assess the suitability of a dataset for reasoning distillation on a student model. DMC provides an assessment by jointly considering data quality, relative difficulty, and student capability. We validated the effectiveness of DMC from two perspectives: (1) DMC exhibits a strong correlation with reasoning distillation performance; and (2) using DMC as the criterion for data selection leads to improved reasoning distillation performance. Both findings are consistently demonstrated across multiple student models and tasks. Moreover, since the DMC of each dataset dynamically changes during training, our experiments demonstrate that dynamically selecting datasets based on DMC can further enhance performance.