TC4D: Trajectory-Conditioned Text-to-4D Generation
This work addresses the gap in realism between 4D and video generation for applications in dynamic 3D scene synthesis, though it appears incremental as it builds on existing text-to-video supervision.
The authors tackled the problem of limited motion generation in text-to-4D synthesis by proposing TC4D, which factors motion into global and local components using trajectory-conditioned generation, resulting in significant improvements in realism and motion amount as evaluated through a user study.
Recent techniques for text-to-4D generation synthesize dynamic 3D scenes using supervision from pre-trained text-to-video models. However, existing representations for motion, such as deformation models or time-dependent neural representations, are limited in the amount of motion they can generate-they cannot synthesize motion extending far beyond the bounding box used for volume rendering. The lack of a more flexible motion model contributes to the gap in realism between 4D generation methods and recent, near-photorealistic video generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D generation, which factors motion into global and local components. We represent the global motion of a scene's bounding box using rigid transformation along a trajectory parameterized by a spline. We learn local deformations that conform to the global trajectory using supervision from a text-to-video model. Our approach enables the synthesis of scenes animated along arbitrary trajectories, compositional scene generation, and significant improvements to the realism and amount of generated motion, which we evaluate qualitatively and through a user study. Video results can be viewed on our website: https://sherwinbahmani.github.io/tc4d.