Reinforcement Learning with Human Feedback for Realistic Traffic Simulation
This work addresses the problem of realistic traffic simulation for autonomous vehicle developers, but it is incremental as it builds on existing RLHF methods applied to a new domain.
The paper tackles the challenge of creating realistic traffic models for autonomous vehicle simulation by developing a framework that uses reinforcement learning with human feedback (RLHF) to enhance realism, achieving proficiency in generating scenarios aligned with human preferences as validated on the nuScenes dataset.
In light of the challenges and costs of real-world testing, autonomous vehicle developers often rely on testing in simulation for the creation of reliable systems. A key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge, an aspect that has proven challenging due to the need to balance realism and diversity. This works aims to address this by developing a framework that employs reinforcement learning with human preference (RLHF) to enhance the realism of existing traffic models. This study also identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models. To tackle these issues, we propose using human feedback for alignment and employ RLHF due to its sample efficiency. We also introduce the first dataset for realism alignment in traffic modeling to support such research. Our framework, named TrafficRLHF, demonstrates its proficiency in generating realistic traffic scenarios that are well-aligned with human preferences, as corroborated by comprehensive evaluations on the nuScenes dataset.