A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs
For researchers in reinforcement learning, this provides a more robust algorithm for average reward SMDPs, addressing a known limitation of existing ratio-based approaches.
The paper addresses the problem of optimizing average reward in semi-Markov decision processes (SMDPs) with non-stationary reward and duration distributions. It introduces a modified harmonic mean operator that correctly computes reward rates, leading to model-free learning algorithms that outperform existing methods in empirical tests.
Recent research has revived and amplified interest in algorithms for undiscounted average reward reinforcement learning in infinite-horizon, non-episodic (continuing) tasks. Semi-Markov decision processes (SMDPs) are of particular interest. In SMDPs, discrete actions stochastically generate both rewards and durations, and the objective is to optimize the average reward rate. Existing algorithms approach this by optimizing the ratio of rewards to durations. However, when rewards and durations are non-stationary (in the infinite horizon), this can be incorrect. This paper presents a novel modified harmonic mean operator that correctly computes reward rates even under such conditions. This yields model-free learning algorithms that can work with SMDPs, while maintaining robustness to non-stationary reward and duration distributions over time. We prove theoretical properties of the modified harmonic mean operator, and empirically demonstrate its efficacy in comparison to existing algorithms.