Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
For researchers building Speech LLMs, this work addresses the structural misalignment between speech encoders and LLMs by proposing translation-based pre-training as a principled alignment mechanism.
The paper investigates whether incorporating translation objectives into speech encoder pre-training improves alignment with LLMs, finding that translation-enhanced pre-training yields superior performance across downstream Speech LLM tasks.
Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM. Unlike encoders based on automatic speech recognition, which often produce representations in separate language-specific spaces, LLMs operate within a unified language-agnostic space. A mechanism is required to align the encoder's language-specific representations with the LLM's shared space. We argue that speech translation provides a principled way to achieve this. Unlike monolingual transcription, translation requires the model to bridge different languages and learn language-agnostic representations. We experimentally evaluate the impact of incorporating translation objectives into speech encoder pre-training. Our results demonstrate that translation-enhanced pre-training improves cross-modal integration and leads to superior performance across downstream Speech LLM tasks.