VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumption
This addresses computational inefficiencies in video generation for AI researchers and practitioners, offering a novel approach to reduce token usage while maintaining quality.
The paper tackles the inefficiency of video tokenization in latent diffusion models by proposing a Duration-Proportional Information Assumption and introducing VFRTok, a variable frame rate tokenizer that achieves competitive reconstruction and state-of-the-art generation fidelity using only 1/8 of the tokens compared to existing methods.
Modern video generation frameworks based on Latent Diffusion Models suffer from inefficiencies in tokenization due to the Frame-Proportional Information Assumption. Existing tokenizers provide fixed temporal compression rates, causing the computational cost of the diffusion model to scale linearly with the frame rate. The paper proposes the Duration-Proportional Information Assumption: the upper bound on the information capacity of a video is proportional to the duration rather than the number of frames. Based on this insight, the paper introduces VFRTok, a Transformer-based video tokenizer, that enables variable frame rate encoding and decoding through asymmetric frame rate training between the encoder and decoder. Furthermore, the paper proposes Partial Rotary Position Embeddings (RoPE) to decouple position and content modeling, which groups correlated patches into unified tokens. The Partial RoPE effectively improves content-awareness, enhancing the video generation capability. Benefiting from the compact and continuous spatio-temporal representation, VFRTok achieves competitive reconstruction quality and state-of-the-art generation fidelity while using only 1/8 tokens compared to existing tokenizers. The code and weights are released at: https://github.com/KwaiVGI/VFRTok.