Playing hard exploration games by watching YouTube
This addresses the challenge of sparse rewards in reinforcement learning for AI agents, enabling imitation from real-world video data, though it is incremental in leveraging existing imitation learning concepts.
The paper tackles the problem of sparse rewards in deep reinforcement learning by proposing a two-stage method that uses noisy, unaligned YouTube videos to guide exploration, achieving human-level performance on hard exploration games like Montezuma's Revenge, Pitfall!, and Private Eye without environment rewards.
Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games Montezuma's Revenge, Pitfall! and Private Eye for the first time, even if the agent is not presented with any environment rewards.