Attention2Probability: Attention-Driven Terminology Probability Estimation for Robust Speech-to-Text System
This work addresses the problem of robust terminology handling in speech recognition and translation for users in domain-specific applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of accurately generating domain-specific terms or neologisms in speech-to-text systems by proposing Attention2Probability, a lightweight method that converts cross-attention weights into presence probabilities, achieving maximum recall rates of 92.57% for Chinese and 86.83% for English with 8.71ms latency and improving terminology accuracy by 6-17% in SLMs.
Recent advances in speech large language models (SLMs) have improved speech recognition and translation in general domains, but accurately generating domain-specific terms or neologisms remains challenging. To address this, we propose Attention2Probability: attention-driven terminology probability estimation for robust speech-to-text system, which is lightweight, flexible, and accurate. Attention2Probability converts cross-attention weights between speech and terminology into presence probabilities, and it further employs curriculum learning to enhance retrieval accuracy. Furthermore, to tackle the lack of data for speech-to-text tasks with terminology intervention, we create and release a new speech dataset with terminology to support future research in this area. Experimental results show that Attention2Probability significantly outperforms the VectorDB method on our test set. Specifically, its maximum recall rates reach 92.57% for Chinese and 86.83% for English. This high recall is achieved with a latency of only 8.71ms per query. Intervening in SLMs' recognition and translation tasks using Attention2Probability-retrieved terms improves terminology accuracy by 6-17%, while revealing that the current utilization of terminology by SLMs has limitations.