Replacing Human Audio with Synthetic Audio for On-device Unspoken Punctuation Prediction
This work addresses the need for efficient punctuation prediction in speech-to-text applications, offering a cost-effective solution by replacing expensive human data with synthetic audio.
The paper tackles the problem of unspoken punctuation prediction for English by developing a multi-modal system that uses synthetic audio instead of human recordings, achieving superior performance with a model designed for on-device use.
We present a novel multi-modal unspoken punctuation prediction system for the English language which combines acoustic and text features. We demonstrate for the first time, that by relying exclusively on synthetic data generated using a prosody-aware text-to-speech system, we can outperform a model trained with expensive human audio recordings on the unspoken punctuation prediction problem. Our model architecture is well suited for on-device use. This is achieved by leveraging hash-based embeddings of automatic speech recognition text output in conjunction with acoustic features as input to a quasi-recurrent neural network, keeping the model size small and latency low.