Evaluating Speech-to-Text Systems with PennSound
This provides a comparative benchmark for speech-to-text systems on diverse audio, useful for users in digital humanities and audio processing, but it is incremental as it applies existing methods to new data.
The researchers evaluated multiple speech-to-text systems using a 10-hour benchmark from PennSound, finding Rev.ai as the top performer and Whisper as the best open-source option, with slim performance differences and trade-offs in accuracy and speed.
A random sample of nearly 10 hours of speech from PennSound, the world's largest online collection of poetry readings and discussions, was used as a benchmark to evaluate several commercial and open-source speech-to-text systems. PennSound's wide variation in recording conditions and speech styles makes it a good representative for many other untranscribed audio collections. Reference transcripts were created by trained annotators, and system transcripts were produced from AWS, Azure, Google, IBM, NeMo, Rev.ai, Whisper, and Whisper.cpp. Based on word error rate, Rev.ai was the top performer, and Whisper was the top open source performer (as long as hallucinations were avoided). AWS had the best diarization error rates among three systems. However, WER and DER differences were slim, and various tradeoffs may motivate choosing different systems for different end users. We also examine the issue of hallucinations in Whisper. Users of Whisper should be cautioned to be aware of runtime options, and whether the speed vs accuracy trade off is acceptable.