Evaluating and Improving Continual Learning in Spoken Language Understanding
This work addresses the need for better evaluation metrics in continual learning for spoken language understanding, though it is incremental as it builds on existing methods.
The paper tackled the problem of evaluating continual learning in spoken language understanding by proposing a unified methodology that assesses stability, plasticity, and generalizability, and demonstrated its sensitivity to task ordering and improvements through knowledge distillation.
Continual learning has emerged as an increasingly important challenge across various tasks, including Spoken Language Understanding (SLU). In SLU, its objective is to effectively handle the emergence of new concepts and evolving environments. The evaluation of continual learning algorithms typically involves assessing the model's stability, plasticity, and generalizability as fundamental aspects of standards. However, existing continual learning metrics primarily focus on only one or two of the properties. They neglect the overall performance across all tasks, and do not adequately disentangle the plasticity versus stability/generalizability trade-offs within the model. In this work, we propose an evaluation methodology that provides a unified evaluation on stability, plasticity, and generalizability in continual learning. By employing the proposed metric, we demonstrate how introducing various knowledge distillations can improve different aspects of these three properties of the SLU model. We further show that our proposed metric is more sensitive in capturing the impact of task ordering in continual learning, making it better suited for practical use-case scenarios.