MInCo: Mitigating Information Conflicts in Distracted Visual Model-based Reinforcement Learning
This addresses a specific bottleneck in visual MBRL for robotics, offering incremental improvements in handling distractions.
The paper tackles the problem of information conflicts in visual model-based reinforcement learning, which hinder robust policy learning in the presence of task-irrelevant distractions, and presents MInCo, a method that mitigates these conflicts and outperforms state-of-the-art methods on robotic control tasks with dynamic background distractions.
Existing visual model-based reinforcement learning (MBRL) algorithms with observation reconstruction often suffer from information conflicts, making it difficult to learn compact representations and hence result in less robust policies, especially in the presence of task-irrelevant visual distractions. In this paper, we first reveal that the information conflicts in current visual MBRL algorithms stem from visual representation learning and latent dynamics modeling with an information-theoretic perspective. Based on this finding, we present a new algorithm to resolve information conflicts for visual MBRL, named MInCo, which mitigates information conflicts by leveraging negative-free contrastive learning, aiding in learning invariant representation and robust policies despite noisy observations. To prevent the dominance of visual representation learning, we introduce time-varying reweighting to bias the learning towards dynamics modeling as training proceeds. We evaluate our method on several robotic control tasks with dynamic background distractions. Our experiments demonstrate that MInCo learns invariant representations against background noise and consistently outperforms current state-of-the-art visual MBRL methods. Code is available at https://github.com/ShiguangSun/minco.