APS: Active Pretraining with Successor Features
This work addresses data-efficiency challenges in reinforcement learning for AI agents, though it appears incremental as it builds on existing techniques like variational successor features and entropy maximization.
The paper tackles the problem of unsupervised pretraining in reinforcement learning by introducing APS, a method that maximizes mutual information between tasks and states via a novel lower bound, achieving significant performance improvements on the Atari 100k benchmark.
We introduce a new unsupervised pretraining objective for reinforcement learning. During the unsupervised reward-free pretraining phase, the agent maximizes mutual information between tasks and states induced by the policy. Our key contribution is a novel lower bound of this intractable quantity. We show that by reinterpreting and combining variational successor features~\citep{Hansen2020Fast} with nonparametric entropy maximization~\citep{liu2021behavior}, the intractable mutual information can be efficiently optimized. The proposed method Active Pretraining with Successor Feature (APS) explores the environment via nonparametric entropy maximization, and the explored data can be efficiently leveraged to learn behavior by variational successor features. APS addresses the limitations of existing mutual information maximization based and entropy maximization based unsupervised RL, and combines the best of both worlds. When evaluated on the Atari 100k data-efficiency benchmark, our approach significantly outperforms previous methods combining unsupervised pretraining with task-specific finetuning.