Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning
This work addresses the challenge of policy learning in diverse dynamics for reinforcement learning applications, representing an incremental improvement over existing cross-domain methods.
The paper tackles the problem of inaccessible expert states in cross-dynamics reinforcement learning by proposing a framework that integrates reward maximization with imitation from observation, using F-distance regularization on globally accessible states. The result is a practical algorithm, ASOR, which significantly improves performance in cross-domain policy transfer across multiple benchmarks.
To learn from data collected in diverse dynamics, Imitation from Observation (IfO) methods leverage expert state trajectories based on the premise that recovering expert state distributions in other dynamics facilitates policy learning in the current one. However, Imitation Learning inherently imposes a performance upper bound of learned policies. Additionally, as the environment dynamics change, certain expert states may become inaccessible, rendering their distributions less valuable for imitation. To address this, we propose a novel framework that integrates reward maximization with IfO, employing F-distance regularized policy optimization. This framework enforces constraints on globally accessible states--those with nonzero visitation frequency across all considered dynamics--mitigating the challenge posed by inaccessible states. By instantiating F-distance in different ways, we derive two theoretical analysis and develop a practical algorithm called Accessible State Oriented Policy Regularization (ASOR). ASOR serves as a general add-on module that can be incorporated into various RL approaches, including offline RL and off-policy RL. Extensive experiments across multiple benchmarks demonstrate ASOR's effectiveness in enhancing state-of-the-art cross-domain policy transfer algorithms, significantly improving their performance.