Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback
This addresses the challenge of efficient teaching in reinforcement learning with limited interaction, though it is incremental as it builds on existing inverse reinforcement learning and active learning techniques.
The paper tackles the problem of teaching sequential decision-making tasks via demonstrations when the teacher lacks access to the learner's model and receives limited feedback, proposing an algorithm that combines active learning and inverse reinforcement learning methods, and shows it is effective in a synthetic car driving environment.
We study the problem of teaching via demonstrations in sequential decision-making tasks. In particular, we focus on the situation when the teacher has no access to the learner's model and policy, and the feedback from the learner is limited to trajectories that start from states selected by the teacher. The necessity to select the starting states and infer the learner's policy creates an opportunity for using the methods of inverse reinforcement learning and active learning by the teacher. In this work, we formalize the teaching process with limited feedback and propose an algorithm that solves this teaching problem. The algorithm uses a modified version of the active value-at-risk method to select the starting states, a modified maximum causal entropy algorithm to infer the policy, and the difficulty score ratio method to choose the teaching demonstrations. We test the algorithm in a synthetic car driving environment and conclude that the proposed algorithm is an effective solution when the learner's feedback is limited.