Towards Combining On-Off-Policy Methods for Real-World Applications
This addresses the problem of inefficient training in reinforcement learning for real-world robotics applications, offering a method to unify on- and off-policy techniques, though it appears incremental as it builds on existing methods like PPO and IMPALA.
The paper tackles the challenge of combining on-policy and off-policy reinforcement learning methods by reformulating the policy gradient objective into a perceptron-like loss function, enabling off-policy training without distinguishing between on and off-policy approaches, and demonstrates this on a quadrotor simulator with a policy running at 500 Hz on a microcontroller.
In this paper, we point out a fundamental property of the objective in reinforcement learning, with which we can reformulate the policy gradient objective into a perceptron-like loss function, removing the need to distinguish between on and off policy training. Namely, we posit that it is sufficient to only update a policy $π$ for cases that satisfy the condition $A(\fracπμ-1)\leq0$, where $A$ is the advantage, and $μ$ is another policy. Furthermore, we show via theoretic derivation that a perceptron-like loss function matches the clipped surrogate objective for PPO. With our new formulation, the policies $π$ and $μ$ can be arbitrarily apart in theory, effectively enabling off-policy training. To examine our derivations, we can combine the on-policy PPO clipped surrogate (which we show to be equivalent with one instance of the new reformation) with the off-policy IMPALA method. We first verify the combined method on the OpenAI Gym pendulum toy problem. Next, we use our method to train a quadrotor position controller in a simulator. Our trained policy is efficient and lightweight enough to perform in a low cost micro-controller at a minimum update rate of 500 Hz. For the quadrotor, we show two experiments to verify our method and demonstrate performance: 1) hovering at a fixed position, and 2) tracking along a specific trajectory. In preliminary trials, we are also able to apply the method to a real-world quadrotor.