Extreme Value Monte Carlo Tree Search for Classical Planning
This work provides an incremental improvement for researchers in automated planning by refining bandit algorithms to better handle unbounded cost-to-go estimates.
The paper tackled the problem of applying Monte Carlo Tree Search with Multi-Armed Bandits to classical planning by addressing issues with existing Gaussian bandits and Full Bellman backup, resulting in a new UCB1-Uniform algorithm with proven regret bounds and empirical performance improvements.
Despite being successful in board games and reinforcement learning (RL), Monte Carlo Tree Search (MCTS) combined with Multi Armed Bandits (MABs) has seen limited success in domain-independent classical planning until recently. Previous work (Wissow and Asai 2024) showed that UCB1, designed for bounded rewards, does not perform well as applied to cost-to-go estimates in classical planning, which are unbounded in $\R$, and showed improved performance using a Gaussian reward MAB instead. This paper further sharpens our understanding of ideal bandits for planning tasks. Existing work has two issues: first, Gaussian MABs under-specify the support of cost-to-go estimates as $(-\infty,\infty)$, which we can narrow down. Second, Full Bellman backup (Schulte and Keller 2014), which backpropagates sample max/min, lacks theoretical justification. We use \emph{Peaks-Over-Threashold Extreme Value Theory} to resolve both issues at once, and propose a new bandit algorithm (UCB1-Uniform). We formally prove its regret bound and empirically demonstrate its performance in classical planning.