Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observation
It addresses a theoretical gap in RL for risk-sensitive decision-making under partial observability, which is incremental but with novel analytical tools.
This work tackles the problem of risk-sensitive reinforcement learning in partially observable environments with hindsight observation by introducing a novel POMDP formulation and developing the first provably efficient algorithm, achieving polynomial regret bounds that match or outperform existing results in degenerate cases.
This work pioneers regret analysis of risk-sensitive reinforcement learning in partially observable environments with hindsight observation, addressing a gap in theoretical exploration. We introduce a novel formulation that integrates hindsight observations into a Partially Observable Markov Decision Process (POMDP) framework, where the goal is to optimize accumulated reward under the entropic risk measure. We develop the first provably efficient RL algorithm tailored for this setting. We also prove by rigorous analysis that our algorithm achieves polynomial regret $\tilde{O}\left(\frac{e^{|γ|H}-1}{|γ|H}H^2\sqrt{KHS^2OA}\right)$, which outperforms or matches existing upper bounds when the model degenerates to risk-neutral or fully observable settings. We adopt the method of change-of-measure and develop a novel analytical tool of beta vectors to streamline mathematical derivations. These techniques are of particular interest to the theoretical study of reinforcement learning.