Self-Consistent Model-based Adaptation for Visual Reinforcement Learning
This work addresses the problem of visual distractions in reinforcement learning for researchers and practitioners working on real-world applications, offering an incremental yet effective solution.
The authors tackled the problem of visual distractions in reinforcement learning, achieving improved performance with their Self-Consistent Model-based Adaptation method, which boosts performance across various distractions. Extensive experiments demonstrated the effectiveness of SCMA, exhibiting better sample efficiency.
Visual reinforcement learning agents typically face serious performance declines in real-world applications caused by visual distractions. Existing methods rely on fine-tuning the policy's representations with hand-crafted augmentations. In this work, we propose Self-Consistent Model-based Adaptation (SCMA), a novel method that fosters robust adaptation without modifying the policy. By transferring cluttered observations to clean ones with a denoising model, SCMA can mitigate distractions for various policies as a plug-and-play enhancement. To optimize the denoising model in an unsupervised manner, we derive an unsupervised distribution matching objective with a theoretical analysis of its optimality. We further present a practical algorithm to optimize the objective by estimating the distribution of clean observations with a pre-trained world model. Extensive experiments on multiple visual generalization benchmarks and real robot data demonstrate that SCMA effectively boosts performance across various distractions and exhibits better sample efficiency.