A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding
This work addresses an incremental improvement for multilingual speech representation in SLU, focusing on distant languages.
The paper tackles the problem of multilingual performance loss and lack of specific-semantics training in SAMU-XLSR for Spoken Language Understanding by proposing a dual task learning approach, resulting in improved semantic enrichment and language portability experiments.
Self-Supervised Learning is vastly used to efficiently represent speech for Spoken Language Understanding, gradually replacing conventional approaches. Meanwhile, textual SSL models are proposed to encode language-agnostic semantics. SAMU-XLSR framework employed this semantic information to enrich multilingual speech representations. A recent study investigated SAMU-XLSR in-domain semantic enrichment by specializing it on downstream transcriptions, leading to state-of-the-art results on a challenging SLU task. This study's interest lies in the loss of multilingual performances and lack of specific-semantics training induced by such specialization in close languages without any SLU implication. We also consider SAMU-XLSR's loss of initial cross-lingual abilities due to a separate SLU fine-tuning. Therefore, this paper proposes a dual task learning approach to improve SAMU-XLSR semantic enrichment while considering distant languages for multilingual and language portability experiments.