Position: Good Embodied Reward Models Need Bad Behavior Data
For the embodied AI community, this work highlights a critical data gap that undermines reward model reliability and proposes concrete steps to address it.
This position paper argues that embodied reward models trained primarily on successful behaviors systematically over-reward unsafe, suboptimal, or shortcut behaviors. The authors show that even modest exposure to real 'bad' behavior data improves alignment with human preferences and reduces false positives.
This position paper argues that to obtain reliable embodied reward models, the community must invest in ``bad'' robot data: failed, suboptimal, error-prone, and even hazardous behaviors. While reward models are central to any foundation model's lifecycle, today's embodied reward models are trained primarily on successful behaviors. We analyze three state-of-the-art embodied reward models and find that they systematically over-reward behaviors that real human evaluators would penalize, including unsafe interactions, poor execution, and shortcut strategies that only superficially satisfy tasks. We attribute these failures to a key data gap: the scarcity of negative embodied data which is costly to collect and often filtered out or withheld in existing robotics datasets. Furthermore, we show that even modest exposure to real bad behavior data can improve alignment with human preferences and reduce costly false positives. We therefore call on the embodied AI community to curate and release their bad robot data, build synthetic bad data generation engines, develop more decentralized physical evaluation systems, and design benchmarks for fine-grained embodied reward model evaluations.