GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech
For text-to-speech systems, GRAFT addresses the problem of mispronouncing rare words by providing direct acoustic control per word, offering a practical improvement over existing phoneme-conditioned models.
GRAFT introduces a per-word pronunciation conditioning mechanism for zero-shot text-to-speech that uses a short spoken sample of a target word to improve pronunciation of rare proper nouns, loanwords, and technical terms. In a blind English listening study, it was ranked first for rendering difficult words, and on a five-language benchmark it reduced target-word phoneme error rate by 22-39% over the text-only backbone while preserving speaker similarity and naturalness.
We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling. Existing systems reach high intelligibility and naturalness but inherit the ambiguity of text and mispronounce rare proper nouns, loanwords and technical terms. Even phoneme-conditioned models offer no direct acoustic handle for per-word pronunciation. GRAFT controls the pronunciation of a chosen word from a short spoken sample of it, encoded with the model's own speech tokenizer and bound to the word's position in the prompt. Voice conversion during training-data construction disentangles the hint speaker from the target speaker, so the hint may come from any voice while the output stays in the target voice. In a blind English listening study, human raters rank GRAFT first by a clear margin, judging its rendering of the difficult word closest to a reference recording of that word. On a five-language objective benchmark, GRAFT reduces target-word phoneme error rate by 22-39% over the identical text-only backbone and outperforms competitive open-source zero-shot systems, both phoneme- and text-conditioned, on target-word pronunciation, while preserving speaker similarity and naturalness.