Robust Asymmetric Learning in POMDPs
This work is significant for researchers and practitioners working on imitation learning in POMDPs, as it tackles the critical issue of expert suboptimality under partial observability, leading to safer and more effective learning.
The paper addresses the problem of imitation learning in Partially Observed Markov Decision Processes (POMDPs) where a fully observed expert might provide suboptimal or unsafe advice due to the trainee's partial information. They propose an objective to train the expert to maximize the imitating agent's expected reward and develop an algorithm, adaptive asymmetric DAgger (A2D), which jointly trains both the expert and the agent, resulting in safer and better-performing policies compared to those learned from a fixed expert.
Policies for partially observed Markov decision processes can be efficiently learned by imitating policies for the corresponding fully observed Markov decision processes. Unfortunately, existing approaches for this kind of imitation learning have a serious flaw: the expert does not know what the trainee cannot see, and so may encourage actions that are sub-optimal, even unsafe, under partial information. We derive an objective to instead train the expert to maximize the expected reward of the imitating agent policy, and use it to construct an efficient algorithm, adaptive asymmetric DAgger (A2D), that jointly trains the expert and the agent. We show that A2D produces an expert policy that the agent can safely imitate, in turn outperforming policies learned by imitating a fixed expert.