Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement
This addresses the issue of domain discrepancy in transfer learning for practitioners with limited data, though it is incremental as it builds on existing regularization methods.
The paper tackles the problem of negative transfer in fine-tuning pre-trained models by proposing Target-awareness REpresentation Disentanglement (TRED), which disentangles target-relevant knowledge from the source model to regularize fine-tuning, resulting in stable improvements of over 2% on average compared to standard fine-tuning.
Fine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples. To obtain better generalization, using the starting point as the reference (SPAR), either through weights or features, has been successfully applied to transfer learning as a regularizer. However, due to the domain discrepancy between the source and target task, there exists obvious risk of negative transfer in a straightforward manner of knowledge preserving. In this paper, we propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED), where the relevant knowledge with respect to the target task is disentangled from the original source model and used as a regularizer during fine-tuning the target model. Specifically, we design two alternative methods, maximizing the Maximum Mean Discrepancy (Max-MMD) and minimizing the mutual information (Min-MI), for the representation disentanglement. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average. TRED also outperforms related state-of-the-art transfer learning regularizers such as L2-SP, AT, DELTA, and BSS.