ASPiRe:Adaptive Skill Priors for Reinforcement Learning
This work addresses the problem of efficient skill reuse in reinforcement learning for AI agents, though it is incremental as it builds on prior methods by introducing adaptive combination of multiple skill priors.
The paper tackles the challenge of accelerating reinforcement learning by leveraging prior experience, introducing ASPiRe which learns a library of distinct skill priors from specialized datasets and combines them adaptively for new tasks, resulting in significant acceleration in learning downstream tasks with improvements over competitive baselines.
We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that leverages prior experience to accelerate reinforcement learning. Unlike existing methods that learn a single skill prior from a large and diverse dataset, our framework learns a library of different distinction skill priors (i.e., behavior priors) from a collection of specialized datasets, and learns how to combine them to solve a new task. This formulation allows the algorithm to acquire a set of specialized skill priors that are more reusable for downstream tasks; however, it also brings up additional challenges of how to effectively combine these unstructured sets of skill priors to form a new prior for new tasks. Specifically, it requires the agent not only to identify which skill prior(s) to use but also how to combine them (either sequentially or concurrently) to form a new prior. To achieve this goal, ASPiRe includes Adaptive Weight Module (AWM) that learns to infer an adaptive weight assignment between different skill priors and uses them to guide policy learning for downstream tasks via weighted Kullback-Leibler divergences. Our experiments demonstrate that ASPiRe can significantly accelerate the learning of new downstream tasks in the presence of multiple priors and show improvement on competitive baselines.