Skill-Based Reinforcement Learning with Intrinsic Reward Matching
This addresses the problem of inefficient skill utilization in reinforcement learning for robotics, though it is incremental as it builds on existing skill discovery methods.
The paper tackles the disconnect between unsupervised skill pretraining and downstream task finetuning in reinforcement learning by introducing Intrinsic Reward Matching (IRM), which uses the skill discriminator to match intrinsic and task rewards for optimal skill selection without environment samples, resulting in improved sample-efficiency and effectiveness on robot manipulation benchmarks.
While unsupervised skill discovery has shown promise in autonomously acquiring behavioral primitives, there is still a large methodological disconnect between task-agnostic skill pretraining and downstream, task-aware finetuning. We present Intrinsic Reward Matching (IRM), which unifies these two phases of learning via the $\textit{skill discriminator}$, a pretraining model component often discarded during finetuning. Conventional approaches finetune pretrained agents directly at the policy level, often relying on expensive environment rollouts to empirically determine the optimal skill. However, often the most concise yet complete description of a task is the reward function itself, and skill learning methods learn an $\textit{intrinsic}$ reward function via the discriminator that corresponds to the skill policy. We propose to leverage the skill discriminator to $\textit{match}$ the intrinsic and downstream task rewards and determine the optimal skill for an unseen task without environment samples, consequently finetuning with greater sample-efficiency. Furthermore, we generalize IRM to sequence skills for complex, long-horizon tasks and demonstrate that IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods on both the Fetch tabletop and Franka Kitchen robot manipulation benchmarks.