Robust Subtask Learning for Compositional Generalization
This addresses the challenge of robust subtask learning for compositional generalization in reinforcement learning, though it appears incremental as an adaptation of existing methods.
The paper tackles the problem of training subtask policies for compositional reinforcement learning to maximize worst-case performance across all possible task sequences, formulating it as a two-agent zero-sum game. The proposed algorithms outperform state-of-the-art baselines on two multi-task environments with continuous states and actions.
Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks. Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform each subtask. In this paper, we focus on the problem of training subtask policies in a way that they can be used to perform any task; here, a task is given by a sequence of subtasks. We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance. We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks. We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RL algorithms to our setting and the other is an asynchronous version which enables parallel training of subtask policies. We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform state-of-the-art baselines.