On Reward-Balancing Methods for Reinforcement Learning
This addresses a specific challenge in reinforcement learning for researchers and practitioners, but appears incremental as it builds on existing normalization and control frameworks.
The paper tackles the problem of solving discounted-return reinforcement learning by introducing reward-balancing methods that adjust the reward function to make optimal policies greedy, and demonstrates performance improvements over state-of-the-art methods through simulation studies.
This paper investigates the so-called reward-balancing methods, a novel class of algorithms for solving discounted-return reinforcement learning (RL) problems. These methods consist of iteratively adjusting the reward function to transform the RL problem into an equivalent one in which the optimal policies are greedy. For this procedure, referred to as normalization process, we provide a theoretical analysis of the involved transformations, emphasizing their algebraic structure. Then, we introduce a control-theoretic reformulation, recasting the reward-balancing procedure into an optimal control framework. The approach is further extended to address model uncertainty through stochastic model sampling, yielding normalization guarantees and probabilistic bounds on stochastic fluctuations. Using the proposed optimal control framework within a scenario model predictive control (MPC) setting, we demonstrate, through simulation studies, performance improvements over the current state-of-the-art.