RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
Provides a multidimensional benchmark for evaluating voice AI systems, highlighting the need for profile-based assessment rather than single aggregate scores.
Current voice AI benchmarks fail to capture acoustic information distinguishing spoken language from text. The RW-Voice-EQ Bench evaluates TTS, STS, SU, and ASR across multiple dimensions, revealing that performance is highly dimension-specific and that real-world conditions expose failures not captured by clean-speech benchmarks.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.