Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding
This addresses the problem of continual learning for spoken language understanding systems, which is incremental as it adapts existing methods to a new scenario.
The paper tackles catastrophic forgetting in class-incremental spoken language understanding by proposing sequence-level knowledge distillation methods, showing that seq-KD improves performance metrics and combining it with audio-KD reduces average WER and enhances entity prediction.
The ability to learn new concepts sequentially is a major weakness for modern neural networks, which hinders their use in non-stationary environments. Their propensity to fit the current data distribution to the detriment of the past acquired knowledge leads to the catastrophic forgetting issue. In this work we tackle the problem of Spoken Language Understanding applied to a continual learning setting. We first define a class-incremental scenario for the SLURP dataset. Then, we propose three knowledge distillation (KD) approaches to mitigate forgetting for a sequence-to-sequence transformer model: the first KD method is applied to the encoder output (audio-KD), and the other two work on the decoder output, either directly on the token-level (tok-KD) or on the sequence-level (seq-KD) distributions. We show that the seq-KD substantially improves all the performance metrics, and its combination with the audio-KD further decreases the average WER and enhances the entity prediction metric.