RefTok: Reference-Based Tokenization for Video Generation
This work addresses the problem of inefficient video modeling for applications like video compression and generation, offering a novel solution with substantial performance gains, though it is incremental in advancing tokenization techniques.
The paper tackled the challenge of temporal redundancy in video generation by introducing RefTok, a reference-based tokenization method that encodes and decodes frames conditioned on a reference frame, resulting in significant improvements: it outperformed state-of-the-art tokenizers by an average of 36.7% across metrics and boosted video generation performance by 27.9% compared to larger models.
Effectively handling temporal redundancy remains a key challenge in learning video models. Prevailing approaches often treat each set of frames independently, failing to effectively capture the temporal dependencies and redundancies inherent in videos. To address this limitation, we introduce RefTok, a novel reference-based tokenization method capable of capturing complex temporal dynamics and contextual information. Our method encodes and decodes sets of frames conditioned on an unquantized reference frame. When decoded, RefTok preserves the continuity of motion and the appearance of objects across frames. For example, RefTok retains facial details despite head motion, reconstructs text correctly, preserves small patterns, and maintains the legibility of handwriting from the context. Across 4 video datasets (K600, UCF-101, BAIR Robot Pushing, and DAVIS), RefTok significantly outperforms current state-of-the-art tokenizers (Cosmos and MAGVIT) and improves all evaluated metrics (PSNR, SSIM, LPIPS) by an average of 36.7% at the same or higher compression ratios. When a video generation model is trained using RefTok's latents on the BAIR Robot Pushing task, the generations not only outperform MAGVIT-B but the larger MAGVIT-L, which has 4x more parameters, across all generation metrics by an average of 27.9%.