Initializing Models with Larger Ones
This addresses the challenge of training small models efficiently in resource-constrained settings, leveraging pretrained models, though it is incremental as it builds on existing initialization and knowledge transfer techniques.
The paper tackles the problem of weight initialization for neural networks by introducing weight selection, a method that initializes smaller models using a subset of weights from a pretrained larger model, resulting in enhanced performance and reduced training time for small models.
Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch. However, the growing number of pretrained models now offers new opportunities for tackling this classical problem of weight initialization. In this work, we introduce weight selection, a method for initializing smaller models by selecting a subset of weights from a pretrained larger model. This enables the transfer of knowledge from pretrained weights to smaller models. Our experiments demonstrate that weight selection can significantly enhance the performance of small models and reduce their training time. Notably, it can also be used together with knowledge distillation. Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era. Code is available at https://github.com/OscarXZQ/weight-selection.