LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model
For researchers in computer vision and robotics, LOME provides a scalable framework for photorealistic human-object interaction generation without explicit 3D/4D modeling, enabling applications in AR/VR and robotic training.
LOME introduces an egocentric world model that generates realistic human-object interaction videos conditioned on input images, text prompts, and per-frame actions, achieving high action-following accuracy and generalization to unseen scenarios, outperforming state-of-the-art methods in temporal consistency and motion control.
Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.