Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting
This work addresses the need for more expressive temporal and risk preferences in safety-critical applications, though it is incremental as it builds on distributional RL with a novel discounting framework.
The paper tackles the problem of optimizing risk-sensitive objectives in reinforcement learning by decoupling time preferences from risk measures through flexible discounting, showing that their multi-horizon extension fixes issues in existing methods and validating robustness in experiments.
Distributional reinforcement learning (RL) is a powerful framework increasingly adopted in safety-critical domains for its ability to optimize risk-sensitive objectives. However, the role of the discount factor is often overlooked, as it is typically treated as a fixed parameter of the Markov decision process or tunable hyperparameter, with little consideration of its effect on the learned policy. In the literature, it is well-known that the discounting function plays a major role in characterizing time preferences of an agent, which an exponential discount factor cannot fully capture. Building on this insight, we propose a novel framework that supports flexible discounting of future rewards and optimization of risk measures in distributional RL. We provide a technical analysis of the optimality of our algorithms, show that our multi-horizon extension fixes issues raised with existing methodologies, and validate the robustness of our methods through extensive experiments. Our results highlight that discounting is a cornerstone in decision-making problems for capturing more expressive temporal and risk preferences profiles, with potential implications for real-world safety-critical applications.