Scalable Weight Reparametrization for Efficient Transfer Learning
This addresses the issue of computational inefficiency in transfer learning for practitioners, offering a scalable solution that reduces parameter updates without sacrificing performance.
The paper tackles the problem of inefficient transfer learning by proposing Scalable Weight Reparametrization (SWR), which uses a policy network to control updated parameters, achieving state-of-the-art performance on multi-lingual keyword spotting and ImageNet-to-Sketch benchmarks with zero additional computations and significantly fewer parameters.
This paper proposes a novel, efficient transfer learning method, called Scalable Weight Reparametrization (SWR) that is efficient and effective for multiple downstream tasks. Efficient transfer learning involves utilizing a pre-trained model trained on a larger dataset and repurposing it for downstream tasks with the aim of maximizing the reuse of the pre-trained model. However, previous works have led to an increase in updated parameters and task-specific modules, resulting in more computations, especially for tiny models. Additionally, there has been no practical consideration for controlling the number of updated parameters. To address these issues, we suggest learning a policy network that can decide where to reparametrize the pre-trained model, while adhering to a given constraint for the number of updated parameters. The policy network is only used during the transfer learning process and not afterward. As a result, our approach attains state-of-the-art performance in a proposed multi-lingual keyword spotting and a standard benchmark, ImageNet-to-Sketch, while requiring zero additional computations and significantly fewer additional parameters.