A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling
This work provides a unified, computationally efficient, and parameter-free method for knowledge transfer across different model sizes, which is highly beneficial for researchers and practitioners working with large-scale pre-trained models and aiming for flexible model scaling.
The paper introduces BoT, a unified framework for transferring knowledge between models of different architectural sizes, treating model weights as continuous signals that can be upsampled (Small-to-Large) or downsampled (Large-to-Small) using Discrete Wavelet Transforms. This approach achieved significant pre-training FLOPs savings of up to 67.1% for S2L and 52.8% for L2S, while maintaining state-of-the-art performance on benchmarks like GLUE and SQuAD.
Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small-to-Large (S2L) and Large-to-Small (L2S) scaling as separate, incompatible problems, focusing on parameter synthesis and selection, respectively. This fragmented perspective has resulted in specialized tools, hindering a unified, bidirectional framework. In this paper, we propose BoT (Bidirectional knowledge Transfer), the first size-agnostic framework to unify S2L and L2S scaling. Our core insight is to treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge. This multi-resolution perspective directly casts S2L and L2S scaling as the signal processing operations of upsampling and downsampling, naturally leading to the adoption of the Discrete Wavelet Transform (DWT) and its Inverse (IDWT). BoT leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner. Extensive experiments on DeiT, BERT, and GPT demonstrate significant pre-training FLOPs savings (up to 67.1% for S2L, 52.8% for L2S) and state-of-the-art performance on benchmarks like GLUE and SQuAD.