Unsupervised Behavior Extraction via Random Intent Priors
This work addresses the challenge of leveraging abundant reward-free data for RL, reducing reliance on human supervision and broadening applicability to real-world scenarios.
The paper tackles the problem of exploiting reward-free offline data in reinforcement learning by proposing UBER, an unsupervised method that extracts diverse behaviors using random pseudo-rewards, which enhances sample efficiency for online RL and outperforms baselines.
Reward-free data is abundant and contains rich prior knowledge of human behaviors, but it is not well exploited by offline reinforcement learning (RL) algorithms. In this paper, we propose UBER, an unsupervised approach to extract useful behaviors from offline reward-free datasets via diversified rewards. UBER assigns different pseudo-rewards sampled from a given prior distribution to different agents to extract a diverse set of behaviors, and reuse them as candidate policies to facilitate the learning of new tasks. Perhaps surprisingly, we show that rewards generated from random neural networks are sufficient to extract diverse and useful behaviors, some even close to expert ones. We provide both empirical and theoretical evidence to justify the use of random priors for the reward function. Experiments on multiple benchmarks showcase UBER's ability to learn effective and diverse behavior sets that enhance sample efficiency for online RL, outperforming existing baselines. By reducing reliance on human supervision, UBER broadens the applicability of RL to real-world scenarios with abundant reward-free data.