Continuous QA Learning with Structured Prompts
This work addresses the problem of lifelong learning for QA models, which is crucial for practical applications, but it is incremental as it builds on existing architecture-based methods with novel prompt enhancements.
The paper tackles the challenge of enabling QA models to learn continuously across tasks without requiring task identities at test time or explicit modeling of unseen tasks, by proposing Diana, a dynamic architecture-based lifelong QA model that uses hierarchically organized prompts to capture knowledge at different granularities, achieving state-of-the-art performance, particularly in handling unseen tasks.
QA models with lifelong learning (LL) abilities are important for practical QA applications, and architecture-based LL methods are reported to be an effective implementation for these models. However, it is non-trivial to extend previous approaches to QA tasks since they either require access to task identities in the testing phase or do not explicitly model samples from unseen tasks. In this paper, we propose Diana: a dynamic architecture-based lifelong QA model that tries to learn a sequence of QA tasks with a prompt enhanced language model. Four types of hierarchically organized prompts are used in Diana to capture QA knowledge from different granularities. Specifically, we dedicate task-level prompts to capture task-specific knowledge to retain high LL performances and maintain instance-level prompts to learn knowledge shared across different input samples to improve the model's generalization performance. Moreover, we dedicate separate prompts to explicitly model unseen tasks and introduce a set of prompt key vectors to facilitate knowledge sharing between tasks. Extensive experiments demonstrate that Diana outperforms state-of-the-art lifelong QA models, especially in handling unseen tasks.