SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation
This addresses the problem of generating diverse and physically plausible human motions in virtual environments for applications like animation and robotics, representing a novel integration rather than a foundational breakthrough.
The paper tackles the challenge of simulating stylized human-scene interactions by introducing SIMS, a hierarchical framework that uses retrieval-augmented script generation with LLMs and a physics-based control policy, which significantly outperforms previous methods in experiments.
Simulating stylized human-scene interactions (HSI) in physical environments is a challenging yet fascinating task. Prior works emphasize long-term execution but fall short in achieving both diverse style and physical plausibility. To tackle this challenge, we introduce a novel hierarchical framework named SIMS that seamlessly bridges highlevel script-driven intent with a low-level control policy, enabling more expressive and diverse human-scene interactions. Specifically, we employ Large Language Models with Retrieval-Augmented Generation (RAG) to generate coherent and diverse long-form scripts, providing a rich foundation for motion planning. A versatile multicondition physics-based control policy is also developed, which leverages text embeddings from the generated scripts to encode stylistic cues, simultaneously perceiving environmental geometries and accomplishing task goals. By integrating the retrieval-augmented script generation with the multi-condition controller, our approach provides a unified solution for generating stylized HSI motions. We further introduce a comprehensive planning dataset produced by RAG and a stylized motion dataset featuring diverse locomotions and interactions. Extensive experiments demonstrate SIMS's effectiveness in executing various tasks and generalizing across different scenarios, significantly outperforming previous methods.