Task Transfer and Domain Adaptation for Zero-Shot Question Answering
This addresses the challenge of domain adaptation for natural language processing tasks when labeled data is unavailable, though it is incremental as it builds on existing pretraining and adaptation methods.
The paper tackles the problem of applying machine learning to new domains without labeled data by using supervised pretraining on source-domain data and combining task transfer with domain adaptation for zero-shot question answering. The result is that their approach outperforms Domain-Adaptive Pretraining in 3 out of 4 domains on downstream domain-specific reading comprehension tasks.
Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning methods to new domains, labeled data may not always be available. To address this, we use supervised pretraining on source-domain data to reduce sample complexity on domain-specific downstream tasks. We evaluate zero-shot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation to fine-tune a pretrained model with no labelled data from the target task. Our approach outperforms Domain-Adaptive Pretraining on downstream domain-specific reading comprehension tasks in 3 out of 4 domains.