Knowledge Flow: Improve Upon Your Teachers
This addresses the challenge for machine learning practitioners in efficiently leveraging existing models for new applications, though it is incremental as it builds on prior knowledge transfer techniques.
The paper tackles the problem of selecting and utilizing pre-trained deep neural networks for new tasks by introducing knowledge flow, a method that transfers knowledge from multiple teacher models to a student model with arbitrary architectures and tasks, resulting in improved performance over fine-tuning and other knowledge exchange methods.
A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.