Assessing and Accelerating Coverage in Deep Reinforcement Learning
This work is significant for DRL practitioners and researchers, as it provides a method to assess and improve the reliability of DRL models, especially in safety-critical applications where incomplete coverage could have severe consequences.
This paper addresses the problem of incomplete state space coverage in Deep Reinforcement Learning (DRL) algorithms, which can lead to critical real-world failures. The authors propose Approximate Pseudo-Coverage (APC) to quantify coverage by projecting high-dimensional state spaces onto lower-dimensional manifolds, and they accelerate coverage using an RRT-based exploration-exploitation strategy.
Current deep reinforcement learning (DRL) algorithms utilize randomness in simulation environments to assume complete coverage in the state space. However, particularly in high dimensions, relying on randomness may lead to gaps in coverage of the trained DRL neural network model, which in turn may lead to drastic and often fatal real-world situations. To the best of the author's knowledge, the assessment of coverage for DRL is lacking in current research literature. Therefore, in this paper, a novel measure, Approximate Pseudo-Coverage (APC), is proposed for assessing the coverage in DRL applications. We propose to calculate APC by projecting the high dimensional state space on to a lower dimensional manifold and quantifying the occupied space. Furthermore, we utilize an exploration-exploitation strategy for coverage maximization using Rapidly-Exploring Random Tree (RRT). The efficacy of the assessment and the acceleration of coverage is demonstrated on standard tasks such as Cartpole, highway-env.