SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning
This work addresses the problem of improving the robustness and adaptability of symbolic planning for decision-making agents by integrating it with RL, which is an incremental step towards more robust AI systems.
The paper introduces SPOTTER, a framework that integrates reinforcement learning (RL) with symbolic planning to resolve discrepancies in symbolic planning models. SPOTTER uses RL to discover new planning operators, enabling agents to achieve goals that were initially unreachable, and it outperforms pure-RL approaches.
Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support ("spot") a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori knowledge about the missing planning operator.