Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
This work addresses the challenge of efficient and robust generalization in reinforcement learning, which is incremental but important for advancing toward generally intelligent systems.
The authors tackled the problem of limited task-to-task transfer in reinforcement learning by introducing the Schema Network, an object-oriented generative physics simulator that learns environment dynamics from data. They reported faster learning and better zero-shot generalization compared to Asynchronous Advantage Actor-Critic and Progressive Networks on Breakout variations.
The recent adaptation of deep neural network-based methods to reinforcement learning and planning domains has yielded remarkable progress on individual tasks. Nonetheless, progress on task-to-task transfer remains limited. In pursuit of efficient and robust generalization, we introduce the Schema Network, an object-oriented generative physics simulator capable of disentangling multiple causes of events and reasoning backward through causes to achieve goals. The richly structured architecture of the Schema Network can learn the dynamics of an environment directly from data. We compare Schema Networks with Asynchronous Advantage Actor-Critic and Progressive Networks on a suite of Breakout variations, reporting results on training efficiency and zero-shot generalization, consistently demonstrating faster, more robust learning and better transfer. We argue that generalizing from limited data and learning causal relationships are essential abilities on the path toward generally intelligent systems.