Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization
This addresses the mismatch between written and spoken registers in Japanese for speech synthesis systems, representing an incremental improvement in domain-specific alignment.
The paper tackled the problem of Japanese SpeechLLMs producing written-style outputs unsuitable for speech synthesis by proposing a preference-based alignment approach, resulting in substantial improvement on the SpokenElyza benchmark while maintaining performance on written-style evaluations.
SpeechLLMs typically combine ASR-trained encoders with text-based LLM backbones, leading them to inherit written-style output patterns unsuitable for text-to-speech synthesis. This mismatch is particularly pronounced in Japanese, where spoken and written registers differ substantially in politeness markers, sentence-final particles, and syntactic complexity. We propose a preference-based alignment approach to adapt Japanese SpeechLLMs for speech-worthy outputs: text that is concise, conversational, and readily synthesized as natural speech. To rigorously evaluate this task, we introduce SpokenElyza, a benchmark for Japanese speech-worthiness derived from ELYZA-tasks-100 with auditory verification by native experts. Experiments show that our approach achieves substantial improvement on SpokenElyza while largely preserving performance on the original written-style evaluation. We will release SpokenElyza to support future research on Japanese spoken dialog systems.