X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning
This addresses IRL problems where prior knowledge of constraints is available, improving efficiency and safety for applications like robotics, but it is incremental as it builds on maximum entropy IRL with a novel sampling method.
The paper tackles the problem of Inverse Reinforcement Learning (IRL) with constraints on optimal policies, proposing X-MEN to guarantee convergence and constraint satisfaction. It shows that X-MEN converges faster to optimal policies in navigation tasks and always satisfies multi-state combinatorial constraints.
Inverse Reinforcement Learning (IRL) is a powerful way of learning from demonstrations. In this paper, we address IRL problems with the availability of prior knowledge that optimal policies will never violate certain constraints. Conventional approaches ignoring these constraints need many demonstrations to converge. We propose XOR-Maximum Entropy Constrained Inverse Reinforcement Learning (X-MEN), which is guaranteed to converge to the optimal policy in linear rate w.r.t. the number of learning iterations. X-MEN embeds XOR-sampling -- a provable sampling approach that transforms the #P complete sampling problem into queries to NP oracles -- into the framework of maximum entropy IRL. X-MEN also guarantees the learned policy will never generate trajectories that violate constraints. Empirical results in navigation demonstrate that X-MEN converges faster to the optimal policies compared to baseline approaches and always generates trajectories that satisfy multi-state combinatorial constraints.