Reflective Policy Optimization
This addresses the problem of high data requirements in reinforcement learning for researchers and practitioners, though it appears incremental as an extension of existing on-policy methods.
The paper tackles the sample inefficiency of on-policy reinforcement learning methods like TRPO and PPO by introducing Reflective Policy Optimization (RPO), which uses past and future state-action information to allow agents to introspect and modify actions, resulting in superior sample efficiency in benchmarks.
On-policy reinforcement learning methods, like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand extensive data per update, leading to sample inefficiency. This paper introduces Reflective Policy Optimization (RPO), a novel on-policy extension that amalgamates past and future state-action information for policy optimization. This approach empowers the agent for introspection, allowing modifications to its actions within the current state. Theoretical analysis confirms that policy performance is monotonically improved and contracts the solution space, consequently expediting the convergence procedure. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks, culminating in superior sample efficiency. The source code of this work is available at https://github.com/Edgargan/RPO.