V-LynX: Token Interface Alignment for Video+X LLMs
This work addresses the challenge of efficiently extending Video LLMs to new modalities without heavy modality-specific encoders or paired supervision, offering a practical solution for multimodal AI.
V-LynX introduces a scalable framework for integrating new modalities into Video LLMs by aligning token interfaces using unpaired unimodal data, achieving state-of-the-art performance across audio-visual QA, 3D reasoning, high-frame-rate, and multi-view video understanding.
This study introduces an intriguing phenomenon in Video LLMs: rather than merely translating frames into textual embeddings, Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture. Exploiting this discovery, we propose V-LynX, a scalable framework that integrates novel modalities into Video LLMs by repurposing the internalized interface. Departing from conventional paradigms that necessitate heavy modality-specific encoders or paired supervision, V-LynX employs a lightweight auxiliary pathway in parallel with the frozen vision encoder. Our method integrates new sensory inputs with intrinsic video priors by aligning both attention responses and statistical distributions using unpaired unimodal data sets. This ensures manifold compatibility while preserving the integrity of the Video LLMs. Extensive benchmarks demonstrate that V-LynX achieves SOTA and efficiency across audio-visual QA, 3D reasoning, high-frame-rate, and multi-view video understanding. The code is available at https://github.com/park-jungin/lynx.