Privileged to Predicted: Towards Sensorimotor Reinforcement Learning for Urban Driving
This work addresses the problem of improving sensorimotor RL for autonomous driving, which is incremental as it builds on existing RL methods by focusing on state representation approximations.
The paper tackled the gap between privileged reinforcement learning (RL) agents, which use ground truth environment representations, and sensorimotor RL agents for urban driving by proposing vision-based models to approximate privileged representations from sensor data, identifying key aspects like route generation and stop zone prediction, and highlighting challenges such as distribution mismatch in bird's-eye-view models.
Reinforcement Learning (RL) has the potential to surpass human performance in driving without needing any expert supervision. Despite its promise, the state-of-the-art in sensorimotor self-driving is dominated by imitation learning methods due to the inherent shortcomings of RL algorithms. Nonetheless, RL agents are able to discover highly successful policies when provided with privileged ground truth representations of the environment. In this work, we investigate what separates privileged RL agents from sensorimotor agents for urban driving in order to bridge the gap between the two. We propose vision-based deep learning models to approximate the privileged representations from sensor data. In particular, we identify aspects of state representation that are crucial for the success of the RL agent such as desired route generation and stop zone prediction, and propose solutions to gradually develop less privileged RL agents. We also observe that bird's-eye-view models trained on offline datasets do not generalize to online RL training due to distribution mismatch. Through rigorous evaluation on the CARLA simulation environment, we shed light on the significance of the state representations in RL for autonomous driving and point to unresolved challenges for future research.