Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics
For researchers in computer graphics and embodied AI, this work provides a scalable solution to generate diverse and physically consistent HOI data, reducing reliance on expensive motion capture.
The paper addresses the problem of synthesizing realistic Human-Object Interactions (HOI) by proposing a framework that uses a physics simulator to generate synthetic training data, overcoming the scarcity and diversity limitations of motion capture datasets. The method achieves enhanced generalization to unseen objects and long-horizon generation with greater dynamic diversity and physical plausibility.
Synthesizing realistic Human-Object Interactions (HOI) is critical for creating embodied avatars and functional virtual environments. However, current data-driven approaches primarily rely on motion capture datasets, which are expensive to scale and limited in functional diversity. Models trained with these datasets fail to generalize to unseen objects and maintain physical consistency over long horizons. In this paper, we propose a novel framework that leverages a physics simulator to overcome the data-scarcity bottleneck in HOI generation. Specifically, we propose a scalable pipeline, called \ours, which leverages policies trained with reinforcement learning in a physics simulator for task-oriented data generation and trains a generative model on the augmented dataset for generalizable HOI generation. To seamlessly utilize the synthetic data, we introduce a coarse-to-fine retargeting process that bridges the representation gap between the simplified model used in physics simulator and the standard parametric body models required for generative training. Validated through comprehensive experiments, our method demonstrates enhanced generalization to unseen objects and the capability of long-horizon generation, while exhibiting greater dynamic diversity and physical plausibility.