TGT: Text-Grounded Trajectories for Locally Controlled Video Generation
This addresses the need for more precise and intuitive control over multi-object scenes in video generation, representing a novel method rather than an incremental improvement.
The paper tackles the problem of limited subject composition control in text-to-video generation by introducing Text-Grounded Trajectories (TGT), which uses point trajectories paired with localized text descriptions to control appearance and motion, achieving higher visual quality, more accurate text alignment, and improved motion controllability compared to prior methods.
Text-to-video generation has advanced rapidly in visual fidelity, whereas standard methods still have limited ability to control the subject composition of generated scenes. Prior work shows that adding localized text control signals, such as bounding boxes or segmentation masks, can help. However, these methods struggle in complex scenarios and degrade in multi-object settings, offering limited precision and lacking a clear correspondence between individual trajectories and visual entities as the number of controllable objects increases. We introduce Text-Grounded Trajectories (TGT), a framework that conditions video generation on trajectories paired with localized text descriptions. We propose Location-Aware Cross-Attention (LACA) to integrate these signals and adopt a dual-CFG scheme to separately modulate local and global text guidance. In addition, we develop a data processing pipeline that produces trajectories with localized descriptions of tracked entities, and we annotate two million high quality video clips to train TGT. Together, these components enable TGT to use point trajectories as intuitive motion handles, pairing each trajectory with text to control both appearance and motion. Extensive experiments show that TGT achieves higher visual quality, more accurate text alignment, and improved motion controllability compared with prior approaches. Website: https://textgroundedtraj.github.io.