Combining imagination and heuristics to learn strategies that generalize
This addresses the challenge of generalization in reinforcement learning for AI systems, though it appears incremental as it builds on existing hierarchical and imagination-based approaches.
The paper tackled the problem of deep reinforcement learning's inability to adapt to minor environmental changes by developing a hierarchical model that combines heuristics and imagination, showing it accelerates learning and promotes transfer to novel games in Wythoff's game.
Deep reinforcement learning can match or exceed human performance in stable contexts, but with minor changes to the environment artificial networks, unlike humans, often cannot adapt. Humans rely on a combination of heuristics to simplify computational load and imagination to extend experiential learning to new and more challenging environments. Motivated by theories of the hierarchical organization of the human prefrontal networks, we have developed a model of hierarchical reinforcement learning that combines both heuristics and imagination into a stumbler-strategist network. We test performance of this network using Wythoff's game, a gridworld environment with a known optimal strategy. We show that a heuristic labeling of each position as hot or cold, combined with imagined play, both accelerates learning and promotes transfer to novel games, while also improving model interpretability.