PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policies
This addresses a key challenge in robotics and autonomous systems for researchers and practitioners, offering a novel approach to handle hidden parameters, though it builds incrementally on existing model-based methods.
The paper tackles the problem of performance degradation in sim-to-real transfer due to unobservable hidden parameters in control tasks by introducing PrivilegedDreamer, a model-based reinforcement learning framework with an explicit parameter estimation module, which outperforms state-of-the-art algorithms on five diverse HIP-MDP tasks.
Numerous real-world control problems involve dynamics and objectives affected by unobservable hidden parameters, ranging from autonomous driving to robotic manipulation, which cause performance degradation during sim-to-real transfer. To represent these kinds of domains, we adopt hidden-parameter Markov decision processes (HIP-MDPs), which model sequential decision problems where hidden variables parameterize transition and reward functions. Existing approaches, such as domain randomization, domain adaptation, and meta-learning, simply treat the effect of hidden parameters as additional variance and often struggle to effectively handle HIP-MDP problems, especially when the rewards are parameterized by hidden variables. We introduce Privileged-Dreamer, a model-based reinforcement learning framework that extends the existing model-based approach by incorporating an explicit parameter estimation module. PrivilegedDreamer features its novel dual recurrent architecture that explicitly estimates hidden parameters from limited historical data and enables us to condition the model, actor, and critic networks on these estimated parameters. Our empirical analysis on five diverse HIP-MDP tasks demonstrates that PrivilegedDreamer outperforms state-of-the-art model-based, model-free, and domain adaptation learning algorithms. Additionally, we conduct ablation studies to justify the inclusion of each component in the proposed architecture.