How Transferable are Neural Networks in NLP Applications?
This work tackles the problem of inconsistent transfer learning results in NLP for researchers and practitioners, offering a foundational analysis that is incremental but crucial for the field.
The paper systematically investigates the transferability of neural networks in NLP, addressing inconsistent prior findings and providing a clearer understanding of how transfer learning works in this domain.
Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain. It is particularly important to neural networks, which are very likely to be overfitting. In some fields like image processing, many studies have shown the effectiveness of neural network-based transfer learning. For neural NLP, however, existing studies have only casually applied transfer learning, and conclusions are inconsistent. In this paper, we conduct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP.