PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation
This work addresses a bottleneck in video understanding and generation systems for researchers and practitioners by enhancing cross-modal alignment and zero-shot transfer capabilities.
The paper tackled the problem of poor cross-modal alignment and zero-shot transfer in video tokenizers by introducing PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions, resulting in state-of-the-art video reconstruction, improved text-to-video quality, and new SOTA zero-shot performance on tasks like video segmentation and temporal action localization across ten benchmarks.
Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introduce PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions. PyraTok builds on a pretrained video VAE and a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive video token sequences. To tightly couple visual tokens with language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the token hierarchy. Across ten benchmarks, PyraTok delivers state-of-the-art (SOTA) video reconstruction, consistently improves text-to-video quality, and sets new SOTA zero-shot performance on video segmentation, temporal action localization, and video understanding, scaling robustly to up to 4K/8K resolutions.