Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
For reinforcement learning researchers, this work addresses a fundamental limitation of dynamic programming under non-exponential discounting, offering a new solution where existing methods diverge.
The paper identifies that Bellman recursions fail under non-exponential discounting, which is common in human preferences and survival processes. It proposes PG-DPO, a variational framework using the Pontryagin Maximum Principle with Monte Carlo rollouts, achieving improved accuracy and stability on hyperbolic and survival-discount benchmarks.
Most value-based and actor--critic reinforcement learning methods rely on Bellman-style recursions, yet these recursions collapse under non-exponential discounting common in human preferences and survival processes. We show the breakdown is structural: exponential discounting sits at a fragile intersection of multiplicativity and time homogeneity, and violating either property breaks standard dynamic programming. To overcome this, we propose Pontryagin-Guided Direct Policy Optimization (PG-DPO), a variational framework that abandons recursion and couples the Pontryagin Maximum Principle with Monte Carlo rollouts via an Adjoint-MC projection enforcing pointwise Hamiltonian maximization. Across multi-dimensional hyperbolic and survival-discount benchmarks, PG-DPO improves accuracy and stability where equation-driven solvers and critic-based baselines diverge.