CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes
This addresses the challenge of text-driven human mesh animation for applications in graphics and AI, offering a novel zero-shot stylization approach that improves over prior methods.
The authors tackled the problem of animating 3D human meshes from text prompts by proposing CLIP-Actor, which recommends motion sequences and optimizes mesh style attributes, resulting in plausible and human-recognizable animations with detailed geometry and texture.
We propose CLIP-Actor, a text-driven motion recommendation and neural mesh stylization system for human mesh animation. CLIP-Actor animates a 3D human mesh to conform to a text prompt by recommending a motion sequence and optimizing mesh style attributes. We build a text-driven human motion recommendation system by leveraging a large-scale human motion dataset with language labels. Given a natural language prompt, CLIP-Actor suggests a text-conforming human motion in a coarse-to-fine manner. Then, our novel zero-shot neural style optimization detailizes and texturizes the recommended mesh sequence to conform to the prompt in a temporally-consistent and pose-agnostic manner. This is distinctive in that prior work fails to generate plausible results when the pose of an artist-designed mesh does not conform to the text from the beginning. We further propose the spatio-temporal view augmentation and mask-weighted embedding attention, which stabilize the optimization process by leveraging multi-frame human motion and rejecting poorly rendered views. We demonstrate that CLIP-Actor produces plausible and human-recognizable style 3D human mesh in motion with detailed geometry and texture solely from a natural language prompt.