Provably Efficient Exploration in Inverse Constrained Reinforcement Learning
This work addresses the challenge of improving sampling efficiency in ICRL for real-world applications where constraints must be inferred from expert data, representing an incremental advancement over existing methods.
The paper tackles the problem of inefficient sampling strategies in Inverse Constrained Reinforcement Learning (ICRL) for inferring constraints from expert behaviors, proposing a strategic exploration framework with two algorithms that achieve provably efficient constraint inference, validated empirically across environments with tractable sample complexity.
Optimizing objective functions subject to constraints is fundamental in many real-world applications. However, these constraints are often not readily defined and must be inferred from expert agent behaviors, a problem known as Inverse Constraint Inference. Inverse Constrained Reinforcement Learning (ICRL) is a common solver for recovering feasible constraints in complex environments, relying on training samples collected from interactive environments. However, the efficacy and efficiency of current sampling strategies remain unclear. We propose a strategic exploration framework for sampling with guaranteed efficiency to bridge this gap. By defining the feasible cost set for ICRL problems, we analyze how estimation errors in transition dynamics and the expert policy influence the feasibility of inferred constraints. Based on this analysis, we introduce two exploratory algorithms to achieve efficient constraint inference via 1) dynamically reducing the bounded aggregate error of cost estimations or 2) strategically constraining the exploration policy around plausibly optimal ones. Both algorithms are theoretically grounded with tractable sample complexity, and their performance is validated empirically across various environments.