Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networks
This enables researchers and neural architecture search systems to efficiently reuse knowledge from trained networks without retraining from scratch, though it is incremental as it builds on existing transfer learning paradigms.
The paper tackles the limitation of existing transfer learning methods that only work between networks with identical architectures by introducing DPIAT, a method using dynamic programming to match and transfer parameters across different architectures, resulting in an average 1.6 times improvement in validation accuracy on ImageNet after 50 epochs.
Transfer learning is a popular technique for improving the performance of neural networks. However, existing methods are limited to transferring parameters between networks with same architectures. We present a method for transferring parameters between neural networks with different architectures. Our method, called DPIAT, uses dynamic programming to match blocks and layers between architectures and transfer parameters efficiently. Compared to existing parameter prediction and random initialization methods, it significantly improves training efficiency and validation accuracy. In experiments on ImageNet, our method improved validation accuracy by an average of 1.6 times after 50 epochs of training. DPIAT allows both researchers and neural architecture search systems to modify trained networks and reuse knowledge, avoiding the need for retraining from scratch. We also introduce a network architecture similarity measure, enabling users to choose the best source network without any training.