Provable Multi-Objective Reinforcement Learning with Generative Models
This work provides the first finite-sample analysis for MORL algorithms, which is important for researchers and practitioners working on real-world tasks with multiple objectives.
This paper addresses multi-objective reinforcement learning (MORL) where multiple objectives exist without known relative costs. The authors propose model-based envelop value iteration (EVI), an algorithm that learns a near-optimal value function with polynomial sample complexity and linear convergence speed.
Multi-objective reinforcement learning (MORL) is an extension of ordinary, single-objective reinforcement learning (RL) that is applicable to many real-world tasks where multiple objectives exist without known relative costs. We study the problem of single policy MORL, which learns an optimal policy given the preference of objectives. Existing methods require strong assumptions such as exact knowledge of the multi-objective Markov decision process, and are analyzed in the limit of infinite data and time. We propose a new algorithm called model-based envelop value iteration (EVI), which generalizes the enveloped multi-objective $Q$-learning algorithm in Yang et al., 2019. Our method can learn a near-optimal value function with polynomial sample complexity and linear convergence speed. To the best of our knowledge, this is the first finite-sample analysis of MORL algorithms.