Teacher Guided Training: An Efficient Framework for Knowledge Transfer
This addresses the challenge of efficient knowledge transfer for deploying compact models in limited data or long-tail settings, though it is incremental as it builds on existing distillation and generative model concepts.
The paper tackles the problem of training compact models without needing large datasets by proposing the Teacher Guided Training (TGT) framework, which leverages pretrained generative models to explore the data domain efficiently, resulting in improved accuracy on image classification, text classification, and retrieval tasks.
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling such large models to compact models for efficient deployment also necessitates a large amount of (labeled or unlabeled) training data. In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data. TGT exploits the fact that the teacher has acquired a good representation of the underlying data domain, which typically corresponds to a much lower dimensional manifold than the input space. Furthermore, we can use the teacher to explore input space more efficiently through sampling or gradient-based methods; thus, making TGT especially attractive for limited data or long-tail settings. We formally capture this benefit of proposed data-domain exploration in our generalization bounds. We find that TGT can improve accuracy on several image classification benchmarks as well as a range of text classification and retrieval tasks.