LAMP: Language-Assisted Motion Planning for Controllable Video Generation
This addresses the challenge of specifying complex object dynamics and camera trajectories for cinematic video generation, offering a novel interface for users.
The paper tackles the problem of motion control in video generation by introducing LAMP, a framework that uses large language models to translate natural language descriptions into explicit 3D trajectories for objects and cameras, achieving improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives.
Video generation has achieved remarkable progress in visual fidelity and controllability, enabling conditioning on text, layout, or motion. Among these, motion control - specifying object dynamics and camera trajectories - is essential for composing complex, cinematic scenes, yet existing interfaces remain limited. We introduce LAMP that leverages large language models (LLMs) as motion planners to translate natural language descriptions into explicit 3D trajectories for dynamic objects and (relatively defined) cameras. LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories. We construct a large-scale procedural dataset pairing natural text descriptions with corresponding motion programs and 3D trajectories. Experiments demonstrate LAMP's improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives establishing the first framework for generating both object and camera motions directly from natural language specifications.