Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
For ASR practitioners, JSTIP offers a simple pretraining strategy to improve entity recognition without requiring synthetic data, though the gains are incremental over existing joint training methods.
The paper proposes Joint Speech-Text Interleaved Pretraining (JSTIP) for ASR, which constructs interleaved speech-text sequences to better leverage LLM priors. On 38k hours of ASR data, JSTIP achieves consistent entity accuracy improvements over baselines and matches performance of synthetic speech-text pairs for domain adaptation.
Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We observe that as supervised ASR training data increases, the contribution of LLM priors becomes less evident, and simple speech-text joint training under-utilizes textual knowledge. We therefore propose Joint Speech-Text Interleaved Pretraining (JSTIP), an ASR-oriented pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences within aligned pairs for speech-LLM architectures that accept continuous inputs. Experiments on 38k hours of ASR data show consistent entity accuracy improvement compared to ASR-only and joint speech-text training baselines. JSTIP achieves on-par entity recognition performance using domain transcription text compared to synthetic speech-text pairs, simplifying domain adaptation. Benefiting from textual pretraining and domain text data, JSTIP is competitive with open-source ASR and Speech-LLM systems in medical entity recognition. The zero-shot speech question answering behaviors further suggest that interleaving reduces the speech-text modality gap and preserves the LLM generative prior, which is likely the reason for the entity improvements on the ASR task.