Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation
This work addresses the challenge of deploying safe and generalizable policies in real-world systems, though it appears incremental as it builds on existing diversity-driven approaches.
The paper tackles the problem of reinforcement learning policies failing in downstream tasks due to model mismatch or perturbations by training diverse policies with regulated behavior patterns to improve generalization, achieving improvements over other diversity-driven methods in various environments.
While Reinforcement Learning can achieve impressive results for complex tasks, the learned policies are generally prone to fail in downstream tasks with even minor model mismatch or unexpected perturbations. Recent works have demonstrated that a policy population with diverse behavior characteristics can generalize to downstream environments with various discrepancies. However, such policies might result in catastrophic damage during the deployment in practical scenarios like real-world systems due to the unrestricted behaviors of trained policies. Furthermore, training diverse policies without regulation of the behavior can result in inadequate feasible policies for extrapolating to a wide range of test conditions with dynamics shifts. In this work, we aim to train diverse policies under the regularization of the behavior patterns. We motivate our paradigm by observing the inverse dynamics in the environment with partial state information and propose Diversity in Regulation (DiR) training diverse policies with regulated behaviors to discover desired patterns that benefit the generalization. Considerable empirical results on various variations of different environments indicate that our method attains improvements over other diversity-driven counterparts.