Self-and-Mixed Attention Decoder with Deep Acoustic Structure for Transformer-based LVCSR
This addresses the problem of enhancing acoustic modeling for speech recognition researchers and practitioners, but it appears incremental as it builds on existing Transformer methods.
They tackled improving acoustic representation in Transformer-based large vocabulary continuous speech recognition by proposing a self-and-mixed attention decoder with deep acoustic structure, achieving CERs of 4.8% on dev and 5.1% on test sets on Aishell-1, which are state-of-the-art results.
The Transformer has shown impressive performance in automatic speech recognition. It uses the encoder-decoder structure with self-attention to learn the relationship between the high-level representation of the source inputs and embedding of the target outputs. In this paper, we propose a novel decoder structure that features a self-and-mixed attention decoder (SMAD) with a deep acoustic structure (DAS) to improve the acoustic representation of Transformer-based LVCSR. Specifically, we introduce a self-attention mechanism to learn a multi-layer deep acoustic structure for multiple levels of acoustic abstraction. We also design a mixed attention mechanism that learns the alignment between different levels of acoustic abstraction and its corresponding linguistic information simultaneously in a shared embedding space. The ASR experiments on Aishell-1 shown that the proposed structure achieves CERs of 4.8% on the dev set and 5.1% on the test set, which are the best results obtained on this task to the best of our knowledge.