Pareto Q-Learning with Reward Machines
For reinforcement learning practitioners dealing with multi-objective tasks, PQLRM offers a sample-efficient method to handle non-Markovian reward structures, though it is an incremental combination of existing techniques.
PQLRM combines Pareto Q-Learning with reward machines to efficiently learn multiple Pareto-optimal policies in multi-objective tasks with non-Markovian rewards, converging faster than a naive baseline and producing policies that QRM cannot.
We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs). PQLRM combines Pareto Q-Learning (PQL), which maintains sets of vector-valued Q-estimates to approximate the Pareto front, with enhancements from Q-Learning with Reward Machines (QRM), which exploits the factored automaton structure of the reward signal. This yields a multi-policy algorithm that remains sample-efficient under non-Markovian, RM-encoded rewards. Experimental trials show that PQLRM converges faster than a naive PQL baseline applied to the cross-product MDP and can synthesize Pareto-optimal policies that QRM cannot.