GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning
This work addresses challenges in robot learning for manipulation tasks by improving policy robustness and generalizability, though it is incremental as it builds on existing video generation and reinforcement learning methods.
The paper tackles the problem of fine-grained manipulation in visual reinforcement learning by proposing GenFlowRL, which uses generative object-centric flow to shape rewards from diverse cross-embodiment datasets, achieving superior performance on 10 manipulation tasks in simulation and real-world evaluations.
Recent advances have shown that video generation models can enhance robot learning by deriving effective robot actions through inverse dynamics. However, these methods heavily depend on the quality of generated data and struggle with fine-grained manipulation due to the lack of environment feedback. While video-based reinforcement learning improves policy robustness, it remains constrained by the uncertainty of video generation and the challenges of collecting large-scale robot datasets for training diffusion models. To address these limitations, we propose GenFlowRL, which derives shaped rewards from generated flow trained from diverse cross-embodiment datasets. This enables learning generalizable and robust policies from diverse demonstrations using low-dimensional, object-centric features. Experiments on 10 manipulation tasks, both in simulation and real-world cross-embodiment evaluations, demonstrate that GenFlowRL effectively leverages manipulation features extracted from generated object-centric flow, consistently achieving superior performance across diverse and challenging scenarios. Our Project Page: https://colinyu1.github.io/genflowrl