Relative Policy-Transition Optimization for Fast Policy Transfer
This work addresses policy transfer in reinforcement learning, which is incremental as it builds on existing theoretical results to improve efficiency in adapting policies across environments.
The paper tackles the problem of policy transfer between Markov Decision Processes by introducing a lemma to measure the relativity gap and proposing the RPTO algorithm, which integrates policy and transition optimization to achieve fast transfer, demonstrating effectiveness on MuJoCo continuous control tasks with variant dynamics.
We consider the problem of policy transfer between two Markov Decision Processes (MDPs). We introduce a lemma based on existing theoretical results in reinforcement learning to measure the relativity gap between two arbitrary MDPs, that is the difference between any two cumulative expected returns defined on different policies and environment dynamics. Based on this lemma, we propose two new algorithms referred to as Relative Policy Optimization (RPO) and Relative Transition Optimization (RTO), which offer fast policy transfer and dynamics modelling, respectively. RPO transfers the policy evaluated in one environment to maximize the return in another, while RTO updates the parameterized dynamics model to reduce the gap between the dynamics of the two environments. Integrating the two algorithms results in the complete Relative Policy-Transition Optimization (RPTO) algorithm, in which the policy interacts with the two environments simultaneously, such that data collections from two environments, policy and transition updates are completed in one closed loop to form a principled learning framework for policy transfer. We demonstrate the effectiveness of RPTO on a set of MuJoCo continuous control tasks by creating policy transfer problems via variant dynamics.