TTSDS -- Text-to-Speech Distribution Score
This provides a standardized evaluation method for TTS researchers and developers, though it is incremental as it builds on existing evaluation concepts.
The paper tackles the problem of evaluating Text-to-Speech (TTS) systems by proposing a multi-factor score that measures prosody, speaker identity, and intelligibility to assess how well synthetic speech mirrors real speech, and shows that this score strongly correlates with human evaluations across 35 TTS systems from 2008 to 2024.
Many recently published Text-to-Speech (TTS) systems produce audio close to real speech. However, TTS evaluation needs to be revisited to make sense of the results obtained with the new architectures, approaches and datasets. We propose evaluating the quality of synthetic speech as a combination of multiple factors such as prosody, speaker identity, and intelligibility. Our approach assesses how well synthetic speech mirrors real speech by obtaining correlates of each factor and measuring their distance from both real speech datasets and noise datasets. We benchmark 35 TTS systems developed between 2008 and 2024 and show that our score computed as an unweighted average of factors strongly correlates with the human evaluations from each time period.