Case-Based Inverse Reinforcement Learning Using Temporal Coherence
This addresses the data efficiency problem in Imitation Learning for reinforcement learning domains, though it is incremental as it builds on existing ideas like temporal coherence.
The paper tackles the problem of expensive expert data in Imitation Learning by proposing an algorithm that imitates higher-level strategies rather than actions, using temporal coherence to predict state similarity. The results show it learns near-optimal policies with very little expert data where action-level methods fail.
Providing expert trajectories in the context of Imitation Learning is often expensive and time-consuming. The goal must therefore be to create algorithms which require as little expert data as possible. In this paper we present an algorithm that imitates the higher-level strategy of the expert rather than just imitating the expert on action level, which we hypothesize requires less expert data and makes training more stable. As a prior, we assume that the higher-level strategy is to reach an unknown target state area, which we hypothesize is a valid prior for many domains in Reinforcement Learning. The target state area is unknown, but since the expert has demonstrated how to reach it, the agent tries to reach states similar to the expert. Building on the idea of Temporal Coherence, our algorithm trains a neural network to predict whether two states are similar, in the sense that they may occur close in time. During inference, the agent compares its current state with expert states from a Case Base for similarity. The results show that our approach can still learn a near-optimal policy in settings with very little expert data, where algorithms that try to imitate the expert at the action level can no longer do so.