Transfer Learning with Pre-trained Conditional Generative Models
This addresses a practical limitation in transfer learning for scenarios with restricted data access and mismatched tasks, though it is an incremental improvement over existing generative approaches.
The paper tackles the problem of transfer learning when source and target tasks have non-overlapping labels, source data is unavailable, and architectures differ, by proposing a two-stage method using conditional generative models to synthesize pseudo data. The method outperforms scratch training and knowledge distillation baselines in experiments.
Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.