Maximum Causal Entropy Specification Inference from Demonstrations
This work addresses safe composition and history dependencies in robotics and similar settings, representing an incremental advance in specification inference methods.
The paper tackles the problem of learning Boolean task specifications from demonstrations by adapting maximum causal entropy inverse reinforcement learning to estimate posterior probabilities, resulting in a polynomial-time algorithm from a naive exponential-time approach.
In many settings (e.g., robotics) demonstrations provide a natural way to specify tasks; however, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the tasks, such as rewards or policies, can be safely composed and/or do not explicitly capture history dependencies. Motivated by this deficit, recent works have proposed learning Boolean task specifications, a class of Boolean non-Markovian rewards which admit well-defined composition and explicitly handle historical dependencies. This work continues this line of research by adapting maximum causal entropy inverse reinforcement learning to estimate the posteriori probability of a specification given a multi-set of demonstrations. The key algorithmic insight is to leverage the extensive literature and tooling on reduced ordered binary decision diagrams to efficiently encode a time unrolled Markov Decision Process. This enables transforming a naive exponential time algorithm into a polynomial time algorithm.