Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding
This addresses inference speed issues for speech synthesis applications where latency is critical, representing an incremental improvement.
The paper tackled the problem of high latency in autoregressive speech synthesis models by proposing Speech Speculative Decoding (SSD), which achieved a 1.4x speedup while maintaining high fidelity and naturalness.
Modern autoregressive speech synthesis models leveraging language models have demonstrated remarkable performance. However, the sequential nature of next token prediction in these models leads to significant latency, hindering their deployment in scenarios where inference speed is critical. In this work, we propose Speech Speculative Decoding (SSD), a novel framework for autoregressive speech synthesis acceleration. Specifically, our method employs a lightweight draft model to generate candidate token sequences, which are subsequently verified in parallel by the target model using the proposed SSD framework. Experimental results demonstrate that SSD achieves a significant speedup of 1.4x compared with conventional autoregressive decoding, while maintaining high fidelity and naturalness. Subjective evaluations further validate the effectiveness of SSD in preserving the perceptual quality of the target model while accelerating inference.