GrASP: Gradient-Based Affordance Selection for Planning
This addresses a key bottleneck in scaling planning methods to large-scale RL problems with continuous actions, though it appears incremental as it builds on existing planning and affordance concepts.
The paper tackles the challenge of planning in continuous action spaces for reinforcement learning by introducing a gradient-based method to select a small number of actions or options for tree-search planning, showing that this approach can outperform model-free RL.
Planning with a learned model is arguably a key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing effectively with continuous action spaces when using tree-search planning (e.g., it is not feasible to consider every action even at just the root node of the tree). In this paper we present a method for selecting affordances useful for planning -- for learning which small number of actions/options from a continuous space of actions/options to consider in the tree-expansion process during planning. We consider affordances that are goal-and-state-conditional mappings to actions/options as well as unconditional affordances that simply select actions/options available in all states. Our selection method is gradient based: we compute gradients through the planning procedure to update the parameters of the function that represents affordances. Our empirical work shows that it is feasible to learn to select both primitive-action and option affordances, and that simultaneously learning to select affordances and planning with a learned value-equivalent model can outperform model-free RL.