Deep Surrogate Q-Learning for Autonomous Driving
This work addresses computational and data efficiency challenges for autonomous driving systems, though it appears incremental as it builds on existing off-policy RL methods.
The paper tackles the problem of deep reinforcement learning's inefficiency in adapting to changing environments and high data requirements for autonomous driving, specifically lane-change behavior with a variable number of vehicles, by proposing Surrogate Q-learning and Scene-centric Experience Replay, which reduce required driving time drastically and are evaluated in the SUMO simulator and on the real highD dataset.
Challenging problems of deep reinforcement learning systems with regard to the application on real systems are their adaptivity to changing environments and their efficiency w.r.t. computational resources and data. In the application of learning lane-change behavior for autonomous driving, agents have to deal with a varying number of surrounding vehicles. Furthermore, the number of required transitions imposes a bottleneck, since test drivers cannot perform an arbitrary amount of lane changes in the real world. In the off-policy setting, additional information on solving the task can be gained by observing actions from others. While in the classical RL setup this knowledge remains unused, we use other drivers as surrogates to learn the agent's value function more efficiently. We propose Surrogate Q-learning that deals with the aforementioned problems and reduces the required driving time drastically. We further propose an efficient implementation based on a permutation-equivariant deep neural network architecture of the Q-function to estimate action-values for a variable number of vehicles in sensor range. We show that the architecture leads to a novel replay sampling technique we call Scene-centric Experience Replay and evaluate the performance of Surrogate Q-learning and Scene-centric Experience Replay in the open traffic simulator SUMO. Additionally, we show that our methods enhance real-world applicability of RL systems by learning policies on the real highD dataset.