Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning
This addresses the challenge of generalization in reinforcement learning for AI systems, though it appears incremental as it builds on existing hierarchical planning methods.
The paper tackles the problem of poor generalization in reinforcement learning by proposing Skipper, a model-based framework that uses spatio-temporal abstractions to decompose tasks into subtasks, resulting in significant zero-shot generalization advantages over state-of-the-art hierarchical planning methods.
Inspired by human conscious planning, we propose Skipper, a model-based reinforcement learning framework utilizing spatio-temporal abstractions to generalize better in novel situations. It automatically decomposes the given task into smaller, more manageable subtasks, and thus enables sparse decision-making and focused computation on the relevant parts of the environment. The decomposition relies on the extraction of an abstracted proxy problem represented as a directed graph, in which vertices and edges are learned end-to-end from hindsight. Our theoretical analyses provide performance guarantees under appropriate assumptions and establish where our approach is expected to be helpful. Generalization-focused experiments validate Skipper's significant advantage in zero-shot generalization, compared to some existing state-of-the-art hierarchical planning methods.