An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
This work addresses the throughput bottleneck in multimodal speech language model inference for practitioners needing efficient generation, but the gains are incremental as it adapts existing vLLM infrastructure.
The paper presents a vLLM-based inference pipeline for unified speech understanding and generation that extends autoregressive decoding to handle multi-stream audio token generation, and introduces a Classifier-Free Guidance (CFG) implementation that sustains 80% of non-CFG throughput by co-scheduling paired conditional and unconditional requests.
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.