Decoding the Ear: A Framework for Objectifying Expressiveness from Human Preference Through Efficient Alignment
This addresses the problem of evaluating and enhancing natural expressiveness in speech synthesis for applications like virtual assistants and entertainment, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the lack of a reliable evaluation metric for speech expressiveness in speech-to-speech models by introducing DeEAR, a framework that converts human preference into an objective score, achieving strong alignment with human perception (SRCC = 0.86) and improving model expressive scores from 2.0 to 23.4 on a 100-point scale.
Recent speech-to-speech (S2S) models generate intelligible speech but still lack natural expressiveness, largely due to the absence of a reliable evaluation metric. Existing approaches, such as subjective MOS ratings, low-level acoustic features, and emotion recognition are costly, limited, or incomplete. To address this, we present DeEAR (Decoding the Expressive Preference of eAR), a framework that converts human preference for speech expressiveness into an objective score. Grounded in phonetics and psychology, DeEAR evaluates speech across three dimensions: Emotion, Prosody, and Spontaneity, achieving strong alignment with human perception (Spearman's Rank Correlation Coefficient, SRCC = 0.86) using fewer than 500 annotated samples. Beyond reliable scoring, DeEAR enables fair benchmarking and targeted data curation. It not only distinguishes expressiveness gaps across S2S models but also selects 14K expressive utterances to form ExpressiveSpeech, which improves the expressive score (from 2.0 to 23.4 on a 100-point scale) of S2S models. Demos and codes are available at https://github.com/FreedomIntelligence/ExpressiveSpeech