Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning
This addresses a fundamental bottleneck in scaling deep reinforcement learning to complex environments, representing a new paradigm rather than an incremental improvement.
The paper tackles the problem of exponentially growing state spaces in high-dimensional MDPs that cause computational complexity and policy failure in reinforcement learning, by introducing a novel paradigm using counteractive actions that achieves significant performance increase and sample-efficiency with zero additional computational complexity.
Following the pivotal success of learning strategies to win at tasks, solely by interacting with an environment without any supervision, agents have gained the ability to make sequential decisions in complex MDPs. Yet, reinforcement learning policies face exponentially growing state spaces in high dimensional MDPs resulting in a dichotomy between computational complexity and policy success. In our paper we focus on the agent's interaction with the environment in a high-dimensional MDP during the learning phase and we introduce a theoretically-founded novel paradigm based on experiences obtained through counteractive actions. Our analysis and method provide a theoretical basis for efficient, effective, scalable and accelerated learning, and further comes with zero additional computational complexity while leading to significant acceleration in training. We conduct extensive experiments in the Arcade Learning Environment with high-dimensional state representation MDPs. The experimental results further verify our theoretical analysis, and our method achieves significant performance increase with substantial sample-efficiency in high-dimensional environments.