TTOM: Test-Time Optimization and Memorization for Compositional Video Generation
This addresses the challenge of improving text-to-video alignment for compositional tasks in video generation models, representing an incremental advancement with practical applications.
The paper tackles the problem of compositional video generation where Video Foundation Models struggle with text-image alignment in scenarios involving motion, numeracy, and spatial relations, and introduces TTOM, a training-free framework that optimizes parameters during inference using layout guidance and a memory mechanism, achieving effective results on T2V-CompBench and Vbench benchmarks.
Video Foundation Models (VFMs) exhibit remarkable visual generation performance, but struggle in compositional scenarios (e.g., motion, numeracy, and spatial relation). In this work, we introduce Test-Time Optimization and Memorization (TTOM), a training-free framework that aligns VFM outputs with spatiotemporal layouts during inference for better text-image alignment. Rather than direct intervention to latents or attention per-sample in existing work, we integrate and optimize new parameters guided by a general layout-attention objective. Furthermore, we formulate video generation within a streaming setting, and maintain historical optimization contexts with a parametric memory mechanism that supports flexible operations, such as insert, read, update, and delete. Notably, we found that TTOM disentangles compositional world knowledge, showing powerful transferability and generalization. Experimental results on the T2V-CompBench and Vbench benchmarks establish TTOM as an effective, practical, scalable, and efficient framework to achieve cross-modal alignment for compositional video generation on the fly.