WhisperNER: Unified Open Named Entity and Speech Recognition
This addresses the need for more accurate and informative speech transcriptions by combining NER and ASR, though it appears incremental by building on existing open NER research.
The paper tackles the problem of integrating named entity recognition with automatic speech recognition to enhance transcription accuracy and informativeness, introducing WhisperNER, a model that outperforms baselines on out-of-domain open type NER and supervised finetuning.
Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and informativeness. In this paper, we introduce WhisperNER, a novel model that allows joint speech transcription and entity recognition. WhisperNER supports open-type NER, enabling recognition of diverse and evolving entities at inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This allows us to train WhisperNER on a large number of examples with diverse NER tags. During training, the model is prompted with NER labels and optimized to output the transcribed utterance along with the corresponding tagged entities. To evaluate WhisperNER, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments demonstrate that WhisperNER outperforms natural baselines on both out-of-domain open type NER and supervised finetuning.