Decoupling Representation Learning from Reinforcement Learning
This work addresses a key bottleneck in reinforcement learning for visual domains by improving representation learning efficiency and generalization, though it is incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of reward-driven feature learning limitations in deep reinforcement learning from images by decoupling representation learning from policy learning, introducing an unsupervised learning task called Augmented Temporal Contrast (ATC) that trains encoders to associate temporally separated observations under augmentations, and shows that ATC-trained encoders match or outperform end-to-end RL in most environments, with agents using these encoders outperforming others in benchmarks.
In an effort to overcome limitations of reward-driven feature learning in deep reinforcement learning (RL) from images, we propose decoupling representation learning from policy learning. To this end, we introduce a new unsupervised learning (UL) task, called Augmented Temporal Contrast (ATC), which trains a convolutional encoder to associate pairs of observations separated by a short time difference, under image augmentations and using a contrastive loss. In online RL experiments, we show that training the encoder exclusively using ATC matches or outperforms end-to-end RL in most environments. Additionally, we benchmark several leading UL algorithms by pre-training encoders on expert demonstrations and using them, with weights frozen, in RL agents; we find that agents using ATC-trained encoders outperform all others. We also train multi-task encoders on data from multiple environments and show generalization to different downstream RL tasks. Finally, we ablate components of ATC, and introduce a new data augmentation to enable replay of (compressed) latent images from pre-trained encoders when RL requires augmentation. Our experiments span visually diverse RL benchmarks in DeepMind Control, DeepMind Lab, and Atari, and our complete code is available at https://github.com/astooke/rlpyt/tree/master/rlpyt/ul.