Joint Learning of Pre-Trained and Random Units for Domain Adaptation in Part-of-Speech Tagging
This addresses domain adaptation for low-resource NLP tasks like POS tagging on social media, but it is incremental as it builds on fine-tuning methods.
The paper tackled the problem of domain adaptation in part-of-speech tagging by augmenting pre-trained networks with random units to better learn target-specific patterns, achieving state-of-the-art performances on three datasets for social media texts.
Fine-tuning neural networks is widely used to transfer valuable knowledge from high-resource to low-resource domains. In a standard fine-tuning scheme, source and target problems are trained using the same architecture. Although capable of adapting to new domains, pre-trained units struggle with learning uncommon target-specific patterns. In this paper, we propose to augment the target-network with normalised, weighted and randomly initialised units that beget a better adaptation while maintaining the valuable source knowledge. Our experiments on POS tagging of social media texts (Tweets domain) demonstrate that our method achieves state-of-the-art performances on 3 commonly used datasets.