Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text
This work addresses efficiency and accuracy challenges in streaming speech recognition and translation for real-time applications, offering incremental improvements over existing RNN-T models.
The paper tackled the problem of improving streaming speech-to-text models by proposing CHAT, a chunk-wise attention transducer that processes audio in fixed-size chunks, resulting in up to 46.2% memory reduction, 1.69X faster inference, and up to 6.3% WER reduction for speech recognition.
We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.