Boomerang Distillation Enables Zero-Shot Model Size Interpolation
This provides a simple and efficient method for reducing training costs and enabling flexible adaptation across deployment environments, though it is incremental as it builds on existing distillation and pruning techniques.
The paper tackles the problem of generating fine-grained model families for large language models under diverse constraints by introducing boomerang distillation, which enables zero-shot interpolation of intermediate-sized models without additional training, achieving performance that scales smoothly and often matches or surpasses existing models of the same size.
Large language models (LLMs) are typically deployed under diverse memory and compute constraints. Existing approaches build model families by training each size independently, which is prohibitively expensive and provides only coarse-grained size options. In this work, we identify a novel phenomenon that we call boomerang distillation: starting from a large base model (the teacher), one first distills down to a small student and then progressively reconstructs intermediate-sized models by re-incorporating blocks of teacher layers into the student without any additional training. This process produces zero-shot interpolated models of many intermediate sizes whose performance scales smoothly between the student and teacher, often matching or surpassing pretrained or distilled models of the same size. We further analyze when this type of interpolation succeeds, showing that alignment between teacher and student through pruning and distillation is essential. Boomerang distillation thus provides a simple and efficient way to generate fine-grained model families, dramatically reducing training cost while enabling flexible adaptation across deployment environments. The code and models are available at https://github.com/dcml-lab/boomerang-distillation.