Learning Logic Specifications for Soft Policy Guidance in POMCP
This work addresses the need for automated policy guidance in POMDPs, reducing reliance on manual expert input, though it is incremental as it builds on existing logic specification methods.
The paper tackles the problem of sparse rewards in Partially Observable Monte Carlo Planning (POMCP) by using inductive logic programming to learn logic specifications from execution traces, which are then integrated to provide soft policy guidance. In benchmark scenarios like rocksample and battery, this approach improved performance with fewer Monte Carlo simulations and scaled to larger instances.
Partially Observable Monte Carlo Planning (POMCP) is an efficient solver for Partially Observable Markov Decision Processes (POMDPs). It allows scaling to large state spaces by computing an approximation of the optimal policy locally and online, using a Monte Carlo Tree Search based strategy. However, POMCP suffers from sparse reward function, namely, rewards achieved only when the final goal is reached, particularly in environments with large state spaces and long horizons. Recently, logic specifications have been integrated into POMCP to guide exploration and to satisfy safety requirements. However, such policy-related rules require manual definition by domain experts, especially in real-world scenarios. In this paper, we use inductive logic programming to learn logic specifications from traces of POMCP executions, i.e., sets of belief-action pairs generated by the planner. Specifically, we learn rules expressed in the paradigm of answer set programming. We then integrate them inside POMCP to provide soft policy bias toward promising actions. In the context of two benchmark scenarios, rocksample and battery, we show that the integration of learned rules from small task instances can improve performance with fewer Monte Carlo simulations and in larger task instances. We make our modified version of POMCP publicly available at https://github.com/GiuMaz/pomcp_clingo.git.