The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models
This work addresses the scaling dynamics of knowledge distillation for code reasoning in LLMs, providing insights for efficient model training, though it is incremental as it builds on existing distillation methods.
The study investigates how the performance of small non-reasoning LLMs scales with the quantity of distillation data for competitive coding skills, revealing a 'valley of code reasoning' where performance initially drops and then increases sharply, and finds that easier questions benefit models more in low-data regimes while output correctness does not affect outcomes.
Distilling the thinking traces of a Large Language Model (LLM) with reasoning capabilities into a smaller model has been proven effective. Yet, there is a scarcity of work done on how model performances scale with the quantity of distillation data. In this work, we study the scaling trend of distilling competitive coding skills on two small non-reasoning LLMs. We validate the hypothesis that there is a $\textit{valley of code reasoning}$: downstream performance on competitive coding first drops as data quantity increases, then it steadily increases in a sharper-than-log-linear fashion. Having identified the trend, we further fine-tune the models at two different distillation stages on the same data to ground conclusions on their respective learning phases. We learn that across stages in the low and medium-low data regimes, small models benefit significantly from easier coding questions than from harder ones. We also find that, surprisingly, the correctness of outputs in training data makes no difference to distillation outcomes. Our work represents a step forward in understanding the training dynamics of code reasoning distillation outside intuition