AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversification
This work addresses a key bottleneck in skill-based reinforcement learning for environments with sparse rewards, offering an incremental improvement over existing methods.
The paper tackles the challenge of simultaneously optimizing exploration and skill diversity in skill-based reinforcement learning, proposing AMPED which uses gradient-surgery projection and a skill selector to achieve performance surpassing baselines across benchmarks and reduce fine-tuning sample complexity with greater diversity.
Skill-based reinforcement learning (SBRL) enables rapid adaptation in environments with sparse rewards by pretraining a skill-conditioned policy. Effective skill learning requires jointly maximizing both exploration and skill diversity. However, existing methods often face challenges in simultaneously optimizing for these two conflicting objectives. In this work, we propose a new method, Adaptive Multi-objective Projection for balancing Exploration and skill Diversification (AMPED), which explicitly addresses both: during pre-training, a gradient-surgery projection balances the exploration and diversity gradients, and during fine-tuning, a skill selector exploits the learned diversity by choosing skills suited to downstream tasks. Our approach achieves performance that surpasses SBRL baselines across various benchmarks. Through an extensive ablation study, we identify the role of each component and demonstrate that each element in AMPED is contributing to performance. We further provide theoretical and empirical evidence that, with a greedy skill selector, greater skill diversity reduces fine-tuning sample complexity. These results highlight the importance of explicitly harmonizing exploration and diversity and demonstrate the effectiveness of AMPED in enabling robust and generalizable skill learning. Project Page: https://geonwoo.me/amped/