Ghost Policies: A New Paradigm for Understanding and Learning from Failure in Deep Reinforcement Learning
This addresses the critical gap in reliable deployment of DRL agents in real-world applications by transforming failures into actionable learning resources.
The paper tackles the problem of opaque failure modes in deep reinforcement learning agents by introducing Ghost Policies and the Arvolution framework, which visualizes failed policy trajectories in augmented reality to enable intuitive understanding and learning from failures, resulting in a new research field called Failure Visualization Learning.
Deep Reinforcement Learning (DRL) agents often exhibit intricate failure modes that are difficult to understand, debug, and learn from. This opacity hinders their reliable deployment in real-world applications. To address this critical gap, we introduce ``Ghost Policies,'' a concept materialized through Arvolution, a novel Augmented Reality (AR) framework. Arvolution renders an agent's historical failed policy trajectories as semi-transparent ``ghosts'' that coexist spatially and temporally with the active agent, enabling an intuitive visualization of policy divergence. Arvolution uniquely integrates: (1) AR visualization of ghost policies, (2) a behavioural taxonomy of DRL maladaptation, (3) a protocol for systematic human disruption to scientifically study failure, and (4) a dual-learning loop where both humans and agents learn from these visualized failures. We propose a paradigm shift, transforming DRL agent failures from opaque, costly errors into invaluable, actionable learning resources, laying the groundwork for a new research field: ``Failure Visualization Learning.''