Efficient Risk-sensitive Planning via Entropic Risk Measures
This work addresses the computational bottleneck in risk-sensitive planning for decision-making applications, offering an efficient method to approximate widely used but costly metrics.
The paper tackles the problem of efficiently approximating risk-sensitive metrics like threshold probabilities or Conditional Value at Risk in Markov Decision Processes, showing that computing optimal policies for Entropic Risk Measures across parameters provides tight approximations with strong empirical performance in various scenarios.
Risk-sensitive planning aims to identify policies maximizing some tail-focused metrics in Markov Decision Processes (MDPs). Such an optimization task can be very costly for the most widely used and interpretable metrics such as threshold probabilities or (Conditional) Values at Risk. Indeed, previous work showed that only Entropic Risk Measures (EntRM) can be efficiently optimized through dynamic programming, leaving a hard-to-interpret parameter to choose. We show that the computation of the full set of optimal policies for EntRM across parameter values leads to tight approximations for the metrics of interest. We prove that this optimality front can be computed effectively thanks to a novel structural analysis and smoothness properties of entropic risks. Empirical results demonstrate that our approach achieves strong performance in a variety of decision-making scenarios.