A Demonstration of Issues with Value-Based Multiobjective Reinforcement Learning Under Stochastic State Transitions
This highlights a critical limitation for researchers and practitioners using these methods in stochastic multiobjective decision-making scenarios, though it is incremental as it focuses on demonstrating an issue rather than proposing a new solution.
The paper identifies a previously unknown problem where model-free, value-based multiobjective reinforcement learning methods fail to find the optimal policy for maximizing Scalarised Expected Return in environments with stochastic state transitions, instead converging to inferior Pareto-dominated solutions.
We report a previously unidentified issue with model-free, value-based approaches to multiobjective reinforcement learning in the context of environments with stochastic state transitions. An example multiobjective Markov Decision Process (MOMDP) is used to demonstrate that under such conditions these approaches may be unable to discover the policy which maximises the Scalarised Expected Return, and in fact may converge to a Pareto-dominated solution. We discuss several alternative methods which may be more suitable for maximising SER in MOMDPs with stochastic transitions.