OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
This addresses efficiency issues for researchers and practitioners working with multimodal AI models, though it is incremental as it builds on existing token compression methods.
The paper tackles the computational bottleneck in processing audio-video tokens for omnimodal large language models by introducing OmniZip, a training-free framework that compresses tokens using audio guidance, achieving a 3.42X inference speedup and 1.4X memory reduction while maintaining performance.
Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding, wherein processing audio-video token sequences creates a significant computational bottleneck, however. Existing token compression methods have yet to accommodate this emerging need of jointly compressing multimodal tokens. To bridge this gap, we present OmniZip, a training-free, audio-guided audio-visual token-compression framework that optimizes multimodal token representation and accelerates inference. Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density, thereby dynamically guiding video token pruning and preserving cues from audio anchors enhanced by cross-modal similarity. For each time window, OmniZip compresses the video tokens using an interleaved spatio-temporal scheme. Extensive empirical results demonstrate the merits of OmniZip - it achieves 3.42X inference speedup and 1.4X memory reduction over other top-performing counterparts, while maintaining performance with no training.