Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech
This provides a more holistic and efficient automatic evaluation metric for synthetic speech, addressing a bottleneck in speech synthesis research, though it is incremental as it builds on existing WER-based approaches.
The paper tackles the challenge of evaluating synthetic speech quality by proposing a method that trains an automatic speech recognition (ASR) model on synthetic speech and tests it on real speech, using Word Error Rate (WER) to measure distribution similarity, which shows strong correlation with human Mean Opinion Scores (MOS) for naturalness and intelligibility across three TTS systems.
Modern speech synthesis systems have improved significantly, with synthetic speech being indistinguishable from real speech. However, efficient and holistic evaluation of synthetic speech still remains a significant challenge. Human evaluation using Mean Opinion Score (MOS) is ideal, but inefficient due to high costs. Therefore, researchers have developed auxiliary automatic metrics like Word Error Rate (WER) to measure intelligibility. Prior works focus on evaluating synthetic speech based on pre-trained speech recognition models, however, this can be limiting since this approach primarily measures speech intelligibility. In this paper, we propose an evaluation technique involving the training of an ASR model on synthetic speech and assessing its performance on real speech. Our main assumption is that by training the ASR model on the synthetic speech, the WER on real speech reflects the similarity between distributions, a broader assessment of synthetic speech quality beyond intelligibility. Our proposed metric demonstrates a strong correlation with both MOS naturalness and MOS intelligibility when compared to SpeechLMScore and MOSNet on three recent Text-to-Speech (TTS) systems: MQTTS, StyleTTS, and YourTTS.