Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm
This work addresses multi-objective optimization in reinforcement learning for engineering problems, but it is incremental as it extends existing policy gradient techniques to a specific formulation.
The paper tackles the problem of maximizing a non-linear concave function of multiple long-term objectives in reinforcement learning by proposing a policy-gradient based model-free algorithm with a biased gradient estimator. It achieves convergence to within ε of the global optima with a sample complexity of O(M⁴σ²/((1-γ)⁸ε⁴)), matching the ε-dependence of standard policy gradient methods.
Many engineering problems have multiple objectives, and the overall aim is to optimize a non-linear function of these objectives. In this paper, we formulate the problem of maximizing a non-linear concave function of multiple long-term objectives. A policy-gradient based model-free algorithm is proposed for the problem. To compute an estimate of the gradient, a biased estimator is proposed. The proposed algorithm is shown to achieve convergence to within an $ε$ of the global optima after sampling $\mathcal{O}(\frac{M^4σ^2}{(1-γ)^8ε^4})$ trajectories where $γ$ is the discount factor and $M$ is the number of the agents, thus achieving the same dependence on $ε$ as the policy gradient algorithm for the standard reinforcement learning.