End-to-end Joint Punctuated and Normalized ASR with a Limited Amount of Punctuated Training Data
This addresses the problem of generating accurate transcripts with punctuation and casing for ASR systems, which is incremental as it builds on existing methods to handle data scarcity.
The paper tackles the challenge of joint punctuated and normalized automatic speech recognition (ASR) with limited punctuated data by proposing two training approaches, resulting in up to 42% relative PC-WER reduction compared to Whisper-base and feasibility with only 5% punctuated data.
Joint punctuated and normalized automatic speech recognition (ASR) aims at outputing transcripts with and without punctuation and casing. This task remains challenging due to the lack of paired speech and punctuated text data in most ASR corpora. We propose two approaches to train an end-to-end joint punctuated and normalized ASR system using limited punctuated data. The first approach uses a language model to convert normalized training transcripts into punctuated transcripts. This achieves a better performance on out-of-domain test data, with up to 17% relative Punctuation-Case-aware Word Error Rate (PC-WER) reduction. The second approach uses a single decoder conditioned on the type of output. This yields a 42% relative PC-WER reduction compared to Whisper-base and a 4% relative (normalized) WER reduction compared to the normalized output of a punctuated-only model. Additionally, our proposed model demonstrates the feasibility of a joint ASR system using as little as 5% punctuated training data with a moderate (2.42% absolute) PC-WER increase.