Rethinking Discrete Speech Representation Tokens for Accent Generation
This work addresses the challenge of accent-controlled speech generation for applications in speech synthesis and voice conversion, representing an incremental advance by systematically investigating and improving upon existing DSRT designs.
The paper tackled the problem of how accent information is encoded in Discrete Speech Representation Tokens (DSRTs) for speech generation, revealing that accent information is substantially reduced with ASR supervision and proposing new DSRT designs that significantly outperform existing ones in controllable accent generation.
Discrete Speech Representation Tokens (DSRTs) have become a foundational component in speech generation. While prior work has extensively studied phonetic and speaker information in DSRTs, how accent information is encoded in DSRTs remains largely unexplored. In this paper, we present the first systematic investigation of accent information in DSRTs. We propose a unified evaluation framework that measures both accessibility of accent information via a novel Accent ABX task and recoverability via cross-accent Voice Conversion (VC) resynthesis. Using this framework, we analyse DSRTs derived from a variety of speech encoders. Our results reveal that accent information is substantially reduced when ASR supervision is used to fine-tune the encoder, but cannot be effectively disentangled from phonetic and speaker information through naive codebook size reduction. Based on these findings, we propose new content-only and content-accent DSRTs that significantly outperform existing designs in controllable accent generation. Our work highlights the importance of accent-aware evaluation and provides practical guidance for designing DSRTs for accent-controlled speech generation.