Learning to Count Words in Fluent Speech enables Online Speech Recognition
This work addresses the latency issue in practical speech recognition applications, offering an online solution that is competitive with offline methods, though it is incremental in nature.
The authors tackled the problem of enabling online speech recognition with low latency by introducing Taris, a Transformer-based system that uses incremental word counting to dynamically segment speech for eager decoding. The system achieved performance comparable to offline models with a dynamic algorithmic delay of 5 segments on datasets like LRS2, LibriSpeech, and Aishell-1.
Sequence to Sequence models, in particular the Transformer, achieve state of the art results in Automatic Speech Recognition. Practical usage is however limited to cases where full utterance latency is acceptable. In this work we introduce Taris, a Transformer-based online speech recognition system aided by an auxiliary task of incremental word counting. We use the cumulative word sum to dynamically segment speech and enable its eager decoding into words. Experiments performed on the LRS2, LibriSpeech, and Aishell-1 datasets of English and Mandarin speech show that the online system performs comparable with the offline one when having a dynamic algorithmic delay of 5 segments. Furthermore, we show that the estimated segment length distribution resembles the word length distribution obtained with forced alignment, although our system does not require an exact segment-to-word equivalence. Taris introduces a negligible overhead compared to a standard Transformer, while the local relationship modelling between inputs and outputs grants invariance to sequence length by design.