Benchmarking Constraint Inference in Inverse Reinforcement Learning
This work addresses the problem of benchmarking constraint inference for researchers in ICRL, though it is incremental as it builds on existing ICRL methods by adding a benchmark and a variational approach.
The authors tackled the lack of common benchmarks in Inverse Constrained Reinforcement Learning (ICRL) by constructing a benchmark in domains like robot control and autonomous driving, and proposed a variational ICRL method to model constraint distributions, facilitating research challenges through extensive experiments.
When deploying Reinforcement Learning (RL) agents into a physical system, we must ensure that these agents are well aware of the underlying constraints. In many real-world problems, however, the constraints are often hard to specify mathematically and unknown to the RL agents. To tackle these issues, Inverse Constrained Reinforcement Learning (ICRL) empirically estimates constraints from expert demonstrations. As an emerging research topic, ICRL does not have common benchmarks, and previous works tested algorithms under hand-crafted environments with manually-generated expert demonstrations. In this paper, we construct an ICRL benchmark in the context of RL application domains, including robot control, and autonomous driving. For each environment, we design relevant constraints and train expert agents to generate demonstration data. Besides, unlike existing baselines that learn a deterministic constraint, we propose a variational ICRL method to model a posterior distribution of candidate constraints. We conduct extensive experiments on these algorithms under our benchmark and show how they can facilitate studying important research challenges for ICRL. The benchmark, including the instructions for reproducing ICRL algorithms, is available at https://github.com/Guiliang/ICRL-benchmarks-public.