Performance of Transfer Learning Model vs. Traditional Neural Network in Low System Resource Environment
This addresses the problem of high resource consumption in transfer learning for NLP applications, but it appears incremental as it focuses on comparing existing methods.
The paper compared the performance and cost of a lighter transfer learning model versus a purpose-built neural network for NLP tasks like text classification and NER in low-resource environments, finding that the lighter model used less computing resources with smaller training data.
Recently, the use of pre-trained model to build neural network based on transfer learning methodology is increasingly popular. These pre-trained models present the benefit of using less computing resources to train model with smaller amount of training data. The rise of state-of-the-art models such as BERT, XLNet and GPT boost accuracy and benefit as a base model for transfer leanring. However, these models are still too complex and consume many computing resource to train for transfer learning with low GPU memory. We will compare the performance and cost between lighter transfer learning model and purposely built neural network for NLP application of text classification and NER model.