Online Risk-Averse Planning in POMDPs Using Iterated CVaR Value Function
This work addresses risk aversion in decision-making under uncertainty for applications like robotics or finance, though it is incremental as it adapts existing algorithms to a new risk measure.
The paper tackled risk-sensitive planning in partially observable Markov decision processes (POMDPs) by extending online planning algorithms to optimize the Iterated Conditional Value-at-Risk (ICVaR) value function, achieving lower tail risk in experiments compared to risk-neutral methods.
We study risk-sensitive planning under partial observability using the dynamic risk measure Iterated Conditional Value-at-Risk (ICVaR). A policy evaluation algorithm for ICVaR is developed with finite-time performance guarantees that do not depend on the cardinality of the action space. Building on this foundation, three widely used online planning algorithms--Sparse Sampling, Particle Filter Trees with Double Progressive Widening (PFT-DPW), and Partially Observable Monte Carlo Planning with Observation Widening (POMCPOW)--are extended to optimize the ICVaR value function rather than the expectation of the return. Our formulations introduce a risk parameter $α$, where $α= 1$ recovers standard expectation-based planning and $α< 1$ induces increasing risk aversion. For ICVaR Sparse Sampling, we establish finite-time performance guarantees under the risk-sensitive objective, which further enable a novel exploration strategy tailored to ICVaR. Experiments on benchmark POMDP domains demonstrate that the proposed ICVaR planners achieve lower tail risk compared to their risk-neutral counterparts.