Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks
This addresses the overlooked exploration challenge in multitask reinforcement learning, potentially explaining the practical success of myopic methods, though it is incremental in extending existing theory.
The paper tackles the problem of inefficient exploration in multitask reinforcement learning by showing that with a diverse set of tasks, a generic policy-sharing algorithm using myopic exploration like ε-greedy can be sample-efficient, as validated in synthetic robotic control experiments.
Multitask Reinforcement Learning (MTRL) approaches have gained increasing attention for its wide applications in many important Reinforcement Learning (RL) tasks. However, while recent advancements in MTRL theory have focused on the improved statistical efficiency by assuming a shared structure across tasks, exploration--a crucial aspect of RL--has been largely overlooked. This paper addresses this gap by showing that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design like $ε$-greedy that are inefficient in general can be sample-efficient for MTRL. To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL. It may also shed light on the enigmatic success of the wide applications of myopic exploration in practice. To validate the role of diversity, we conduct experiments on synthetic robotic control environments, where the diverse task set aligns with the task selection by automatic curriculum learning, which is empirically shown to improve sample-efficiency.