Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning
This addresses a practical challenge in imitation learning for real-world applications where acquiring expert observations is costly, though it is incremental as it builds on existing methods like importance weighting and learning with rejection.
The paper tackles the problem of imitation learning when the demonstrator and learner have different observation spaces and limited overlap, proposing the IWRE algorithm to address dynamics and support mismatches. Experimental results show IWRE successfully handles tasks like converting vision-based demonstrations to RAM-based policies in Atari, even with limited visual observations.
In many real-world imitation learning tasks, the demonstrator and the learner have to act under different observation spaces. This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces. On the other hand, previous studies under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process. However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations. In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL). We identify two underlying issues in HOIL: the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE) algorithm based on importance weighting and learning with rejection to solve HOIL problems. Experimental results show that IWRE can solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations to random access memory (RAM)-based policies in the Atari domain, even with limited visual observations.